<jupyter_start><jupyter_text>Lithium-ion BatteriesThis case study is based upon the excellent example below:https://apmonitor.com/pds/index.php/Main/LithiumIonBatteriesAlong with its github repository:https://github.com/APMonitor/pdsand license:https://github.com/APMonitor/pds/blob/main/LICENSEIn particular, as described on the web page above:Physical and chemical properties of the Lithium-ion silicate cathodes are used to predict the crystal structure of a Lithium-ion battery material as monoclinic, orthorhombic and triclinic. This case study demonstrates how feature engineering improves the classification results.![](http://cdn.pixabay.com/photo/2020/01/28/15/18/battery-4800010_960_720.jpg)**Background:** [Lithium-ion batteries](https://en.wikipedia.org/wiki/Lithium-ion_battery) are commonly used for portable electronics, electric vehicles, and aerospace applications. During discharge, Lithium ions move from the negative electrode through an electrolyte to the positive electrode to create a voltage and current. During recharging, the ions migrate back to the negative electrode. The crystal structure (monoclinic, orthorhombic, triclinic) is available for 339 different chemicals that contain lithium. Available properties are:| Label | Description ||------|------||   Materials Id  | Unique ID of the material from materialsproject.org ||   Formula  | Chemical formula of the compound ||   Space group  | Symmetry group of a three-dimensional crystal pattern ||   Formation Energy (eV)  | Energy required to produce the material from standard elements ||   E Above Hull (eV)  | Energy released if compound is decomposed into the most stable compounds ||   Band Gap (eV)  | Energy range in a solid where no electronic states can exist. A larger band gap indicates that the compound is worse at conducting electricity or heat. ||   Nsites  | Number of atoms in the unit cell of the crystal ||   Density (gm/cc)  | Mass per volume of bulk crystalline materials ||   Volume  | Unit cell volume of the material ||   Has Bandstructure  | Boolean (True / False) variable for bandstructure |**Objective:** Predict the crystal structure type (monoclinic, orthorhombic, triclinic) from Lithium-ion physical and chemical compound information.**References**<NAME>. (2021, December 16). Lithium-ion Batteries, Machine Learning for Engineers. Retrieved from https://apmonitor.com/pds/index.php/Main/LithiumIonBatteries  Import Polymer MFR Data<jupyter_code># install package
# using for variable encoding

try:
    import chemparse
except:
    !pip install chemparse
    print('May need to restart kernel to use chemparse')<jupyter_output><empty_output><jupyter_text>Similar to case study 1, the data can be accessed directly by url.<jupyter_code>import pandas as pd
url = 'http://apmonitor.com/pds/uploads/Main/lithium_ion.txt'
data = pd.read_csv(url)
data.sample(20)<jupyter_output><empty_output><jupyter_text>Still remember the ValueError we met in Case Study 1? Because we forgot to check if there are any null values in the dataset. This time, let's do check that first.![](https://i.pinimg.com/originals/05/ba/0a/05ba0af7235165ec94078d0d6661047f.jpg)**Recall:**data.isnull() is used to check if there is null value in the dataset. We can sum them by using sum().data.info() and data.describe() are usually used to check the properties and basic statistics of the data.<jupyter_code>data.isnull().sum().sum()
data.info()
data.describe()<jupyter_output><empty_output><jupyter_text>Great!!! This dataset does not contain any null values. 🙂 Let's start modeling.  Naive solution Classification ModelingWhat is the first step? Splitting the training and test set right? **Recall:**The `train_test_split` is a function in `sklearn` for the specific purpose of splitting data into train and test sets.<jupyter_code>from sklearn.model_selection import train_test_split
train,test = train_test_split(data, test_size=0.2, shuffle=True, random_state=1)<jupyter_output><empty_output><jupyter_text>Not sure which column could be a useful feature? Let's start with the naive solution use them all first, except for `Materials Id`, since `Materials Id` is not important in this case.**Features：**Formula	Spacegroup, Formation Energy (eV), 	E Above Hull (eV), 	Band Gap (eV), 	Nsites, 	Density (gm/cc), 	Volume, 	Has Bandstructure**label：** Crystal System<jupyter_code>from sklearn.tree import DecisionTreeClassifier
method = DecisionTreeClassifier()
model = method.fit(train[['Formula', 'Spacegroup', 'Formation Energy (eV)', 'E Above Hull (eV)', 'Band Gap (eV)', 'Nsites', 'Density (gm/cc)', 'Volume', 'Has Bandstructure']].to_numpy(),train['Crystal System'].to_numpy())<jupyter_output><empty_output><jupyter_text>ValueError: could not convert string to float: 'Li4Fe2SiO7'????![?](https://www.mememaker.net/static/images/memes/4379873.jpg) What is 'Li4Fe2SiO7'? It looks like a Formula. Why we neet to convert it to float?Wow! Our model can not handle the string information directly, so we need to convert it to value. Let's check their data type first. `dtypes` is used to check data's type.<jupyter_code>data.dtypes<jupyter_output><empty_output><jupyter_text>It turns out that we need to distinguish the types of variables first. 👀  **Question 1** **(15 points)****Part 1 (5 points)**What is the differnce between interval (also called numerical), ordinal and categorical variables? **Part 2 (5 points)**  Why does it matter whether a variable is categorical, ordinal or interval? Give an example.**Part 3 (5 points)**  Identifies the data type of the following variables in the dataset.`Materials Id`              `Formula`                  `Spacegroup`                `Formation Energy (eV)`    `E Above Hull (eV)`        `Band Gap (eV)`            `Nsites`                     `Density (gm/cc)`          `Volume`                   `Has Bandstructure`           `Crystal System`            Hint: Review Randy's 🧙 class notes that can be found from Canvas.<jupyter_code># Your answer here
# Your answer here
# Your answer here
# Your answer here
# Your answer here
# Separate into numerical features that don't need preprocessing, and categorical features that need to be transformed
num_feat = data.select_dtypes(include=['int64','float64']).columns
cat_feat = data.select_dtypes(include=['object','bool']).columns
num_feat
cat_feat<jupyter_output><empty_output><jupyter_text>Still remember how to use heatmap and pair plot to show the correlation between variables? Let's discover the correlation of all numerical variables.**Recall:**sns.heatmap(cor, annot=True,cmap=plt.cm.Reds)sns.pairplot(data)<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
import plotly.express as px
plt.figure(figsize=(10,8))
cor = data[num_feat].corr()
sns.heatmap(cor, annot=True,cmap=plt.cm.Reds)
plt.show()
sns.pairplot(data[num_feat])<jupyter_output><empty_output><jupyter_text>**Question 2** **(15 points)****Part 1** (5 points)Find the two **most** positively correlated(relevant) variables. Why do you think they are relevant? **Part 2** (5 points)Find the two **least** correlated (cloasest to 0) variables. Why do you think they are **not** relevant? **Part 3** (5 points)What about negative correlations?  Are two variables that are very negatively correlated predictable from each other?<jupyter_code># Your answer here
# Your answer here
# Your answer here
# Your answer here
# Your answer here<jupyter_output><empty_output><jupyter_text>Okay, now I know which variables are numerical and which are categorical. Since categorical ones bring ValueErrors, can we just use numerical ones? Let's give it a try.<jupyter_code>method = DecisionTreeClassifier()
model = method.fit(train[num_feat].to_numpy(),train['Crystal System'].to_numpy())<jupyter_output><empty_output><jupyter_text>Great!!! The model was successfully fit！ 🙂 Now, we can evaluate the performance on test set.<jupyter_code>from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score
predicted_label=model.predict(test[num_feat].to_numpy())
groundtruth_label=test['Crystal System'].to_numpy()

accuracy=accuracy_score(predicted_label,groundtruth_label)

macro_precision=precision_score(predicted_label,groundtruth_label,average='macro')
micro_precision=precision_score(predicted_label,groundtruth_label,average='micro')

macro_recall=recall_score(predicted_label,groundtruth_label,average='macro')
micro_recall=recall_score(predicted_label,groundtruth_label,average='micro')

macro_f1=f1_score(predicted_label,groundtruth_label,average='macro')
micro_f1=f1_score(predicted_label,groundtruth_label,average='micro')

from tabulate import tabulate
results = [
["accuracy", accuracy],
["macro_precision", macro_precision],
["micro_precision",micro_precision],
["macro_recall",macro_recall],
["micro_recall",micro_recall],
["macro_f1",macro_f1],
["micro_f1",micro_f1],
] 
print (tabulate(results, headers=["Metrics", "Values"]))<jupyter_output>Metrics            Values
---------------  --------
accuracy         0.529412
macro_precision  0.507215
micro_precision  0.529412
macro_recall     0.515152
micro_recall     0.529412
macro_f1         0.507768
micro_f1         0.529412<jupyter_text>Ok, we already see things are interesting. There are several different metrics (accuracy, precision, recall, f1)! Hmmm....Here is a great tutorial to help you out.https://androidkt.com/micro-macro-averages-for-imbalance-multiclass-classification/:~:text=Macro%20average%20deals%20with%20aggregates%20or%20totals%2C%20macro,price%20levels%2C%20and%20gross%20domestic%20product%20or%20GDP.Part of this class is learning to be able to use outside resources to learn more about these topics.  Here is a chance to get some practice 😀  **Question 3** **(10 points)****Part 1 (5 points)**What do **original (binary)** accuracy, precision, recall and f1 mean? How are they counted?**Part 2 (5 points)**  What do **macro/micro** precision, recall and f1 mean? How are they counted? Why we use macro/micro metrics here instead of original binery ones?<jupyter_code># Your answer here
# Your answer here
# Your answer here
# Your answer here
# Your answer here<jupyter_output><empty_output><jupyter_text>Solution with Feature Engineering![?](https://www.mememaker.net/static/images/memes/4751028.jpg) Randy 🧙 wanna ask you a question since you dropped all the categorical variables. Are these variables really not useful for classification? Let's encode these variables to allow them to be fit into the model. Then, we can see if they can impact the performance of the task.  Categorical encoding methods 1. One Hot EncodingMethod: Encode each category value into a binary vector, with size =  of distinct values. See https://towardsdatascience.com/understanding-feature-engineering-part-2-categorical-data-f54324193e63Example: Has Bandstructure column has 2 distinct values, True and False. Create a new column where 1 = True and 0 = False.Pros: simple and rugged method to get categorical features into unique and useful numerical featuresCons: m unique values results in m unique new features. This is fine when there are only 2-3 unique values (such as hi/lo, yes/no), but creates issues when there are more. Can't handle new categories that weren't in training data, and easily overfit. Sparse data. 2. Encode to ordinal variablesMethod: assign each unique value to a unique number. Example: Spacegroup = Pc is assigned to 0, Spacegroup = P21/c is assigned to 1, etc. Pros: simple and quick, 1 column in -> 1 column outCons: residual "structure" (number assigned is arbitrary, and it leads algorithms to assume that a Spacegroup with a value of 20 is higher value than a Spacegroup of value 1) 3. Feature HashingMethod: Encode each unique category into a non-binary vectorExample: Spacegroup = Pc is assigned to [1,0,0], Spacegroup = P21/c is assigned to [1,2,-1], etc. Specify number of columns (length of vector)Pros: low dimensionality so really efficient.Cons: potential collisions (for example the 1st value in example has both Spacegroups sharing a '1'); hashed features aren't interpretable so can't be used in feature importance. **How do you come up with hashes to preserve distances!?** 4. Other methodsPrimarily involve prior knowledge about dataset. Encode with own algorithm to include closely related features.Variation on One Hot Encoding for large numbers of unique values: classify infrequent instances into "rare" category. May lose some granularity and important info, but also allows for new categories that aren't in training data Now, let's explore categorical variables one by one.<jupyter_code>cat_feat<jupyter_output><empty_output><jupyter_text>`'Materials Id'` column<jupyter_code>data['Materials Id'].describe()<jupyter_output><empty_output><jupyter_text>339 unique values for 339 unique entries; there is no useful information in this column and it can be dropped.  **Question 4** **(5 points)**Why can this column be dropped?  I.e., why is a column with all unique values (and no prior information) not useful?<jupyter_code>data.drop(columns=['Materials Id'],inplace=True)
data.columns
# Your answer here
# Your answer here
# Your answer here
# Your answer here
# Your answer here<jupyter_output><empty_output><jupyter_text>`'Has Bandstructure'` column<jupyter_code>data['Has Bandstructure'].value_counts().plot(kind='bar')<jupyter_output><empty_output><jupyter_text>2 unique values, True and False. Classic example of when to use one-hot encoding<jupyter_code># One-hot encode 'Has Bandstructure'
data['Has Bandstructure'] = data['Has Bandstructure'].map({True:1, False:0})<jupyter_output><empty_output><jupyter_text>`'Spacegroup'` column<jupyter_code>data['Spacegroup'].value_counts().plot(kind='bar')
print(data['Spacegroup'].nunique())<jupyter_output>44<jupyter_text>44 unique values, with most of them occuring multiple times**Option 1:** One-hot encoding will result in 44 new feature columns; inefficient and memory-intensive.**Option 2:** Encode to ordinal numbers. Will possibly work, but does leave a residual structure that may affect model performance**Option 3:** Use Feature Hashing to create a vector representation of each unique Spacegroup. Note that if vector size = 44, it's the same as one-hot encoding, and if vector size = 1, it's the same as encoding to ordinal variables. What other vector size might be useful?  3 or 4?<jupyter_code># Option 1: One-hot encoding (not used)
pd.get_dummies(data['Spacegroup'])
# Option 2: Ordinal number encoding
data['Spacegroup (ordinal)'] = pd.factorize(data['Spacegroup'])[0]

# Can also order by most common to least common
data['Spacegroup'].rank(method="dense").astype(int)
# Option 3: Feature Hashing
from sklearn.feature_extraction import FeatureHasher
n = 3
fh = FeatureHasher(n_features=n, input_type='string')
hashed_tag = fh.fit_transform(data['Spacegroup']).toarray()
ht_df = pd.DataFrame(hashed_tag)
ht_df.columns = ['Spacegroup'+str(i) for i in range(n)]

data = data.join(ht_df)<jupyter_output><empty_output><jupyter_text>`'Formula'` column<jupyter_code>data['Formula'].value_counts()<jupyter_output><empty_output><jupyter_text>114 unique values, most only occuring once. One-hot encoding is out of the question**Option 1,2,3:** one-hot encoding, ordinal number encoding, and feature hashing all become inefficient with such variety.**Option 4:** Use domain knowledge to create additional features. For example, we can look at the LiFeSiO4 formula, and turn it into 4 new columns, each one indicating how many of each atom are in the formula (for example, {Li: 1, Fe: 1, Si: 1, O: 4}) 🧙  THIS MAKES RANDY HAPPY! ![face](https://wwwcache.wralsportsfan.com/asset/voices/2017/08/24/16902022/excited_baby-DMID1-5bvd6d0n1-480x293.gif)<jupyter_code># Option 4: use chemparse package to create the new features of atom counts
import chemparse
chem_data = data['Formula'].apply(chemparse.parse_formula)

# Convert the dictionary into a dataframe and fill NaN's with zero's
chem_data = pd.json_normalize(chem_data)
chem_data = chem_data.fillna(0)

data = data.join(chem_data)<jupyter_output><empty_output><jupyter_text>`'Crystal System'` columnThis is the target column, and there are 3 different types of crystal structures we're trying to classify. To properly transform this to numerical data, we have to understand if we are working on a multi-class problem or a multi-label problem.- A multi-class problem is one in which there is only one distinct type of classification for each row. For example, a fruit is either an apple or an orange, but cannot be both. For a multi-class problem, the target value should be a single value, such as a 0 for apple and 1 for orange. In other words, it would be encoded to ordinal numbers.- A multi-label problem is one in which there are possibly multiple labels for each row. For example, classifying pictures of apples and oranges can include a picture of an apple alone, an orange alone, or both an apple and an orange. For a multi-label problem, the target value should be a vector representation, such as [1,0] for apple, [0,1] for orange, and [1,1] for both apple and orange. In other words, we would have to one-hot encode the target feature.Since the crystal system structure is unique, this is a multi-class problem. The target output should be encoded to a 0, 1, or 2. If it were a multi-label problem, the target output would have to be encoded to a vector of length 3.<jupyter_code># Check for balance
data['Crystal System'].value_counts().plot(kind='bar')
# Encode Crystal System to ordinal values for multi-class problem
data['Crystal System (#)'] = data['Crystal System'].map({'monoclinic':0, 'orthorhombic':1, 'triclinic':2})<jupyter_output><empty_output><jupyter_text>Test performance<jupyter_code># All new numerical features (Crystal System excluded, since it's int32)
features = list(data.select_dtypes(include=['int64','float64']).columns.values)

ord_feat = ['Formation Energy (eV)','E Above Hull (eV)','Band Gap (eV)',
            'Nsites','Density (gm/cc)','Volume','Has Bandstructure',
            'Spacegroup (ordinal)','Li','Mn','Si','O','Fe','Co'
           ]

hash_feat = ['Formation Energy (eV)','E Above Hull (eV)','Band Gap (eV)',
             'Nsites','Density (gm/cc)','Volume','Has Bandstructure',
             'Spacegroup0','Spacegroup1','Spacegroup2',
             'Li','Mn','Si','O','Fe','Co'
            ]

labels = ['Crystal System (#)']
titles = ['Original Numerical Features Only',
          'With Encoded Features\n(Ordinal Spacegroup)',
          'With Encoded Features\n(Hashed Spacegroup)'
         ]

results=[]

for i, feat in enumerate([num_feat,ord_feat,hash_feat]):
    X = data[feat]
    y = data[labels]

    # 80% training data and 20% testing
    Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.2,shuffle=True, random_state=1)
    dtree = DecisionTreeClassifier()
    dtree.fit(Xtrain,ytrain)
    yp = dtree.predict(Xtest)

    accuracy=accuracy_score(yp,ytest)

    macro_precision=precision_score(yp,ytest,average='macro')
    micro_precision=precision_score(yp,ytest,average='micro')

    macro_recall=recall_score(yp,ytest,average='macro')
    micro_recall=recall_score(yp,ytest,average='micro')

    macro_f1=f1_score(yp,ytest,average='macro')
    micro_f1=f1_score(yp,ytest,average='micro')

    results.append([titles[i],accuracy,macro_precision,micro_precision,macro_recall,micro_recall,macro_f1,micro_f1])

print (tabulate(results, headers=["features", "accuracy", "macro_precision", "micro_precision", "macro_recall", "micro_recall", "macro_f1", "micro_f1"],tablefmt="grid"))<jupyter_output>+----------------------------------+------------+-------------------+-------------------+----------------+----------------+------------+------------+
| features                         |   accuracy |   macro_precision |   micro_precision |   macro_recall |   micro_recall |   macro_f1 |   micro_f1 |
+==================================+============+===================+===================+================+================+============+============+
| Original Numerical Features Only |   0.514706 |          0.491943 |          0.514706 |       0.499012 |       0.514706 |   0.489496 |   0.514706 |
+----------------------------------+------------+-------------------+-------------------+----------------+----------------+------------+------------+
| With Encoded Features            |   1        |          1        |          1        |       1        |       1        |   1        |   1        |
| (Ordinal Spacegroup)             |            |                   |                   |           [...]<jupyter_text>Want to check the accuracy of each class? We can use the confusion matrix to show the results.<jupyter_code>from sklearn.metrics import confusion_matrix,plot_confusion_matrix

plt.figure(figsize=(16,5))

titles = ['Original Numerical Features Only',
          'With Encoded Features\n(Ordinal Spacegroup)',
          'With Encoded Features\n(Hashed Spacegroup)'
         ]

results=[]

for i, feat in enumerate([num_feat,ord_feat,hash_feat]):
    X = data[feat]
    y = data[labels]

    # 80% training data and 20% testing
    Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.2,shuffle=True, random_state=1)
    dtree = DecisionTreeClassifier()
    dtree.fit(Xtrain,ytrain)
    yp = dtree.predict(Xtest)

    plt.subplot(1,3,i+1)
    cm = confusion_matrix(ytest,yp)
    sns.heatmap(cm,annot=True)
    plt.title(titles[i])<jupyter_output><empty_output><jupyter_text>**Question 5** **(10 points)**Based on the above metrics table and confusion matrix, which feature engineering technique performs best and why?<jupyter_code># Your answer here
# Your answer here
# Your answer here
# Your answer here
# Your answer here<jupyter_output><empty_output><jupyter_text>**Question 6** **(15 points)**Try two different methods such as `linear_model.LogisticRegression`, `neighbors.KNeighborsClassifier` or `ensemble.RandomForestClassifier`. Report your results.Can you achieve better performance than any of the previous models? Why do these methods perform better/worse than the decision tree?<jupyter_code>'''
Hint:
from sklearn.linear_model import LogisticRegression
for i, feat in enumerate([num_feat,ord_feat,hash_feat]):
    X = data[feat]
    y = data[labels]
    #80% training data and 20% testing
    Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.2,shuffle=True, random_state=1)

    # change to your selected model
    model = LogisticRegression(max_iter=10000)


    model.fit(Xtrain,ytrain)
    yp = model.predict(Xtest)
'''
# Your code and answer here
# Your code and answer here
# Your code and answer here
# Your code and answer here
# Your code and answer here
# Your code and answer here
# Your code and answer here<jupyter_output><empty_output><jupyter_text>Solution with Feature Engineering + rebalance dataset![](https://media.makeameme.org/created/today-rebalance.jpg) Let's review the distribution of `Crystal System` again. It is clear that the three class is unbalanced. Dealing with imbalanced data is common in classification problems because such imbalance usually affects the performance of the model.<jupyter_code># Check for balance
data['Crystal System'].value_counts().plot(kind='bar')<jupyter_output><empty_output><jupyter_text>Here, we use apply undersampling to `monoclinic` and `orthorhombic` so that they have the same number as `triclinic`.<jupyter_code>monoclinic_data=data[data['Crystal System'] == 'monoclinic']
orthorhombic_data=data[data['Crystal System'] == 'orthorhombic']
triclinic_data=data[data['Crystal System'] == 'triclinic']

monoclinic_data_sampled = monoclinic_data.sample(n=len(triclinic_data))
orthorhombic_data_sampled = orthorhombic_data.sample(n=len(triclinic_data))

data_rebalanced=triclinic_data.append(monoclinic_data_sampled).append(orthorhombic_data_sampled).reset_index(drop=True)
# Check for balance
data_rebalanced['Crystal System'].value_counts().plot(kind='bar')<jupyter_output><empty_output><jupyter_text>Now that the dataset has been rebalanced, let's see if this affects the performance of the model.  **Question 7** **(15 points)**Redo the calculations, using the above `data_rebalanced` dataset. Report results and compare the results with the original(imbalance) dataset. Will rebalancing the data improve performance? Why?<jupyter_code>'''
Hint:

for i, feat in enumerate([num_feat,ord_feat,hash_feat]):

    # change to rebalanced dataset: data_rebalanced
    X = data_rebalanced[feat]
    y = data_rebalanced[labels]


    # 80% training data and 20% testing
    Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.2,shuffle=True, random_state=1)
    dtree = DecisionTreeClassifier()
    dtree.fit(Xtrain,ytrain)
    yp = dtree.predict(Xtest)
    
'''
# Your code and answer here
# Your code and answer here
# Your code and answer here
# Your code and answer here
# Your code and answer here
# Your code and answer here
# Your code and answer here<jupyter_output><empty_output><jupyter_text>Solution with Feature Engineering + dimensionality reduction![](https://i.ytimg.com/vi/hSDinYYObd4/maxresdefault.jpg) Remember the mysterious technology Principal Component Analysis (PCA) that Randy 🧙 mentioned the lecture? Forgotten? Go review it, because it may appear on your midterm/final exam. 👀 Now, let's see how PCA works in the classification task. PCA performs better with a normalized feature set. We will perform standard scalar normalization to normalize our feature set first using `StandardScaler` and then apply `PCA`.<jupyter_code>from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
plt.figure(figsize=(16,5))

titles = ['Original Numerical Features Only',
          'With Encoded Features\n(Ordinal Spacegroup)',
          'With Encoded Features\n(Hashed Spacegroup)'
         ]

results=[]

for i, feat in enumerate([num_feat,ord_feat,hash_feat]):
    X = data[feat]
    y = data[labels]

    # 
    sc = StandardScaler()
    pca = PCA(n_components=4)  # adjust the number of principal component here
    X = sc.fit_transform(X)
    X = pca.fit_transform(X)


    # 80% training data and 20% testing
    Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.2,shuffle=True, random_state=1)
    dtree = DecisionTreeClassifier()
    dtree.fit(Xtrain,ytrain)
    yp = dtree.predict(Xtest)

    accuracy=accuracy_score(yp,ytest)

    macro_precision=precision_score(yp,ytest,average='macro')
    micro_precision=precision_score(yp,ytest,average='micro')

    macro_recall=recall_score(yp,ytest,average='macro')
    micro_recall=recall_score(yp,ytest,average='micro')

    macro_f1=f1_score(yp,ytest,average='macro')
    micro_f1=f1_score(yp,ytest,average='micro')

    results.append([titles[i],accuracy,macro_precision,micro_precision,macro_recall,micro_recall,macro_f1,micro_f1])
    
    plt.subplot(1,3,i+1)
    cm = confusion_matrix(ytest,yp)
    sns.heatmap(cm,annot=True)
    plt.title(titles[i])

print (tabulate(results, headers=["features", "accuracy", "macro_precision", "micro_precision", "macro_recall", "micro_recall", "macro_f1", "micro_f1"],tablefmt="grid"))<jupyter_output>+----------------------------------+------------+-------------------+-------------------+----------------+----------------+------------+------------+
| features                         |   accuracy |   macro_precision |   micro_precision |   macro_recall |   micro_recall |   macro_f1 |   micro_f1 |
+==================================+============+===================+===================+================+================+============+============+
| Original Numerical Features Only |   0.573529 |          0.595839 |          0.573529 |       0.581739 |       0.573529 |   0.585761 |   0.573529 |
+----------------------------------+------------+-------------------+-------------------+----------------+----------------+------------+------------+
| With Encoded Features            |   0.661765 |          0.673761 |          0.661765 |       0.662393 |       0.661765 |   0.666247 |   0.661765 |
| (Ordinal Spacegroup)             |            |                   |                   |           [...]<jupyter_text>**Question 8** **(15 points)****Part 1 (5 points)**How does PCA perform dimensionality reduction? Is it valid for numerical variable? Why? Is it valid for ordinal variable? Why? Is it valid for categorical variable? Why?**Part 2 (10 points)**Try different number of principal components (eg. `n_components=1,2,3,4`) on different methods (eg. `LogisticRegression`, `KNeighborsClassifier` or `naive_bayes` ), and report the results.Does the classification effect increase as the number of principal components increases? Why?<jupyter_code># Your code and answer here
# Your code and answer here
# Your code and answer here
# Your code and answer here
# Your code and answer here
# Your code and answer here
# Your code and answer here<jupyter_output><empty_output><jupyter_text>**Question 9** **($\infty$ points, since this is where you really learn stuff! :-)**You can get as many points of extra credit as you like by doing a good job with this question!Play around with doing a better analysis.  Things to try:1.   This problem is easier that Case Study 1 in that you seem to have enough data to do a good job.  Can you make the problem harder and still do well?  E.g., if you make your training data smaller and smaller then is there some limit where you don't do well anymore?  This tells you how many experiments you need!2.   Try different train and test splits and see if the answer changes (i.e., ```train_processed,test_processed = train_test_split(ds, test_size=0.2, shuffle=True, random_state=1234)```3.   Try different dimensionality reduction methods (other than PCA). Here are several examples in Sci-kit learn (https://scikit-learn.org/stable/auto_examples/manifold/plot_compare_methods.html?highlight=manifold%20learning). Do any work better than others?<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12688630533995138, 'fraction_numerical': 0.020353775941903463, 'mean_word_length': 4.538745387453875, 'pattern_counts': {'":': 0, '<': 126, '<?xml version=': 0, '>': 127, 'https://': 14, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '230942', 'license': 'MIT', 'n_tokens_mistral': 9351, 'n_tokens_neox': 8562, 'n_words': 2932, 'path': 'CaseStudy_2_Classification.ipynb', 'repo_name': 'rcpaffenroth/-Machine-Learning-for-Engineering-and-Science-Applications', 'size': '30028'}