<jupyter_start><jupyter_text>2.s986, Wed 17 September, 2020, <NAME>  Useful resource on learning jupyter notebooks functionality<jupyter_code>%%HTML
 <iframe src="https://www.youtube.com/watch?v=HW29067qVWk" title="Jupyter Notebook Tutorial: Introduction, Setup, and Walkthrough"></iframe><jupyter_output><empty_output><jupyter_text>Notes to self:To run bash commands, prefix a bash command with an exclamation mark in a code cell: !pwd \To list magic commands, type %lsmagic \To run commands in different computer languages you can use the magic cell commands \To view user defined variables type magic commans %who \<jupyter_code>!pwd
%lsmagic
%%latex
\noindent
\newline
\bf{Hello World}<jupyter_output><empty_output><jupyter_text>a) Code below imports packages/libraries for managing the data. pandas library is for 'working' with data sets<jupyter_code>import pandas as pd #package for working with data sets
from matplotlib import pyplot as plt #'from library import function', package for plotting/visualizing data
import numpy as np #package for scientific/numerical computing
import seaborn as sb #package for statistical graphics
from sklearn import preprocessing #for normalizing dataset<jupyter_output><empty_output><jupyter_text>b) Code below reads data and stores it in a data type called 'data'<jupyter_code>data = pd.read_excel('Concrete_Data.xls') #This line creates a DataFrame object and stores it in variable 'data'
type(data) #type() spits out the data type such as string, list, etc.<jupyter_output><empty_output><jupyter_text>Code below displays the columns of the data<jupyter_code>data.columns<jupyter_output><empty_output><jupyter_text>Code below to check number(length) of rows and columns<jupyter_code>len(data) #number of rows
len(data.columns) #number of columns<jupyter_output><empty_output><jupyter_text>c) Code below checks how many columns are input variables and output variables<jupyter_code>%%bash
ls
cat Concrete_Readme.txt | grep 'input\|output'<jupyter_output>concrete_compressive_strength_data_curation.ipynb
Concrete_Data.xls
Concrete_Readme.txt
Attribute breakdown: 8 quantitative input variables, and 1 quantitative output variable<jupyter_text>d) Code below eliminates 'Fine Aggregate' from data set<jupyter_code>#del data["Fine Aggregate (component 7)(kg in a m^3 mixture)"]
#data.pop('Fine Aggregate (component 7)(kg in a m^3 mixture)')<jupyter_output><empty_output><jupyter_text>I couldn't get the above commands to work so I tried a different way, below.<jupyter_code>data_columns = ['Cement (component 1)(kg in a m^3 mixture)',
       'Blast Furnace Slag (component 2)(kg in a m^3 mixture)',
       'Fly Ash (component 3)(kg in a m^3 mixture)',
       'Water  (component 4)(kg in a m^3 mixture)',
       'Superplasticizer (component 5)(kg in a m^3 mixture)',
       'Coarse Aggregate  (component 6)(kg in a m^3 mixture)',
       'Fine Aggregate (component 7)(kg in a m^3 mixture)', 'Age (day)',
       'Concrete compressive strength(MPa, megapascals) ']
del data_columns[7]
print(data.columns)<jupyter_output>Index(['Cement (component 1)(kg in a m^3 mixture)',
       'Blast Furnace Slag (component 2)(kg in a m^3 mixture)',
       'Fly Ash (component 3)(kg in a m^3 mixture)',
       'Water  (component 4)(kg in a m^3 mixture)',
       'Superplasticizer (component 5)(kg in a m^3 mixture)',
       'Coarse Aggregate  (component 6)(kg in a m^3 mixture)', 'Age (day)',
       'Concrete compressive strength(MPa, megapascals) '],
      dtype='object')<jupyter_text>e) Normalize data with mean 0 and standard deviation 1 and check that it's normalized<jupyter_code>%who 
data.head() # similar to command: print(data_normalized)
data_normalized = preprocessing.scale(data) # Function normalizes data by columb
data_normalized[:10] # similar to bash command head but only displays number of rows specified after ":"<jupyter_output><empty_output><jupyter_text>Everything from here on is for my own intellectual enlightenment Visualizing the data before and after normalization, was throwing errors at me<jupyter_code>type(data)
type(data_normalized)
plt.title("Data Visualization Demo") 
plt.xlabel("input") 
plt.ylabel("output") 
plt.xlim(-1,1)
plt.ylim(-1,1)
plt.plot(data_normalized[1],data_normalized[len(data_normalized)]) 
plt.show()<jupyter_output><empty_output><jupyter_text>I used the source code from class materials to create scatter plots of the:raw data<jupyter_code>fs = 20

n_col = 4 # num of columns per row in the figure
y_indx = -1 #Compressive Strength [MPa] # Choose which column is the response variable 
color_indx = -2 #Age [days] # Choose which column is shown as the superimposed color
column_names = data.columns.values

cmap = plt.get_cmap('coolwarm', 10)
print("The color is for variable", column_names[color_indx], "with red as the highest and blue as the lowest")

for n in np.arange(0, 8, n_col):
        fig,axes = plt.subplots(1, n_col, figsize=(18, 3.5), sharey = False)
        for i in np.arange(n_col):
            #print(n)
            if n< len(column_names)-1:
                im = axes[i].scatter(data.iloc[:,n],data.iloc[:,y_indx], 
                                     c=data.iloc[:,color_indx], s = 20, cmap=cmap, alpha =0.5, edgecolors = 'face')
                axes[i].set_xlabel(column_names[n])
            else:
                axes[i].axis("off")
                #axes[i].set_title(sf_cols[n])
            n = n+1      
        axes[0].set_ylabel(column_names[y_indx])
        for i in range(len(axes)):
            axes[i].tick_params(direction='in', length=5, width=1, labelsize = fs*.8, grid_alpha = 0.5)
            axes[i].grid(True, linestyle='-.')
        plt.show()<jupyter_output>The color is for variable Age (day) with red as the highest and blue as the lowest<jupyter_text>normalized data<jupyter_code>for n in np.arange(0, 8, n_col):
        fig,axes = plt.subplots(1, n_col, figsize=(18, 3.5), sharey = False)
        for i in np.arange(n_col):
            #print(n)
            if n< len(column_names)-1:
                im = axes[i].scatter(data_normalized.iloc[:,n],data_normalized.iloc[:,y_indx], 
                                     c=data_normalized.iloc[:,color_indx], s = 20, cmap=cmap, alpha =0.5, edgecolors = 'face')
                axes[i].set_xlabel(column_names[n])
            else:
                axes[i].axis("off")
                #axes[i].set_title(sf_cols[n])
            n = n+1      
        axes[0].set_ylabel(column_names[y_indx])
        for i in range(len(axes)):
            axes[i].tick_params(direction='in', length=5, width=1, labelsize = fs*.8, grid_alpha = 0.5)
            axes[i].grid(True, linestyle='-.')
        plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11257091669154971, 'fraction_numerical': 0.015228426395939087, 'mean_word_length': 4.0788476118271415, 'pattern_counts': {'":': 1, '<': 55, '<?xml version=': 0, '>': 53, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '45782', 'license': 'MIT', 'n_tokens_mistral': 2280, 'n_tokens_neox': 2159, 'n_words': 644, 'path': 'Alban repo/Homework 2/Concrete Data Assignment/concrete_compressive_strength_data_curation.ipynb', 'repo_name': 'albancobi/2s986', 'size': '6702'}