<jupyter_start><jupyter_text>Assignment 1: Exploring and Visualizing Data - <NAME>  Purpose The purpose of this exercise is to explore the results from the MSPA Software Survey that was collected in December 2016. As MSDS academic administrators, we are tasked with defining the future direction of the graduate program while incorporating feedback from students. The results of the survey provides us information about the software preferences, interest in new courses and other information relevant to data science curriculum planning. The following report will outline key findings of the MSPA Software Survey and make recommendations on what to incorporate into the graduate program.    Section 1 - Setting up the dataframe<jupyter_code>#To complete this exercise, we need to utilize certain libraries 
#This block of code imports all the ones we will need
#Code Credit: Jump-Start Example: Python analysis of MSPA Software Survey
# Update 2017-09-21 by <NAME> and <NAME>
# Update 2018-06-30 by <NAME> v005 transformation code added

import pandas as pd  
import numpy as np  
import matplotlib.pyplot as plt
import seaborn as sns  
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')
sns.set_style("whitegrid") #background color of the plots
sns.set_context({"figure.figsize": (12, 6.5)}) #size of each plot 
#One of the most popular visuals to show correlation is a heat map 
#This function creates a heat map which will be used in the exercise
def corr_chart(df_corr):
    corr=df_corr.corr()
    top = np.zeros_like(corr, dtype=np.bool)
    top[np.triu_indices_from(top)] = True
    fig=plt.figure()
    fig, ax = plt.subplots(figsize=(12,12))
    sns.heatmap(corr, mask=top, cmap='coolwarm', 
        center = 0, square=True, 
        linewidths=.5, cbar_kws={'shrink':.5}, 
        annot = True, annot_kws={'size': 9}, fmt = '.3f')           
    plt.xticks(rotation=45) 
    plt.yticks(rotation=0) 
    plt.title('Correlation Heat Map')   
    plt.savefig('plot-corr-map.pdf', 
        bbox_inches = 'tight', dpi=None, facecolor='w', edgecolor='b', 
        orientation='portrait', papertype=None, format=None, 
        transparent=True, pad_inches=0.25, frameon=None)      

np.set_printoptions(precision=3)
#This code block loads the data from the MSPA Software Survey
survey_data = pd.read_csv("mspa-survey-data.csv")
#Since each observation is pertaining to a student, we will make
#RespondentID the label for each observation with the following code 
survey_data.set_index('RespondentID', drop = True, inplace = True)
#Using the rename function will change the names of the all the software columns
#We want shorter column names 
survey_df = survey_data.rename(index=str, columns={
    'Personal_JavaScalaSpark': 'My_Java',
    'Personal_JavaScriptHTMLCSS': 'My_JS',
    'Personal_Python': 'My_Python',
    'Personal_R': 'My_R',
    'Personal_SAS': 'My_SAS',
    'Professional_JavaScalaSpark': 'Prof_Java',
    'Professional_JavaScriptHTMLCSS': 'Prof_JS',
    'Professional_Python': 'Prof_Python',
    'Professional_R': 'Prof_R',
    'Professional_SAS': 'Prof_SAS',
    'Industry_JavaScalaSpark': 'Ind_Java',
    'Industry_JavaScriptHTMLCSS': 'Ind_JS',
    'Industry_Python': 'Ind_Python',
    'Industry_R': 'Ind_R',
    'Industry_SAS': 'Ind_SAS'})<jupyter_output><empty_output><jupyter_text>Key Takeaway* This section outlines which libraries were loaded to complete the exercise, loading the CSV file from our working directory, taking a quick look at the dataframe and shortening the names of the columns   Section 2 - Software Variables Analysis<jupyter_code>#First, we will create a new dataframe with just the renamed 
#software variables 
software_df = survey_df.loc[:, "My_Java":"Ind_SAS"]
#This output shows descriptive statistics for software preference variables
print('\nDescriptive statistics for survey data ---------------')
print(software_df.describe())
#The corr_chart function created earlier shows the correlation between 
#all the software preference variables 
corr_chart(software_df)
plt.show()
#Let's study the Industry variables a bit more because the MSDS program
#wants to prepare students for what they believe the industry values

#Using seaborn, let's create barplots of two of the Industry variables 
fig = plt.figure()

ax1 = fig.add_subplot(221)
sns.barplot(x="Ind_Java", y="Graduate_Date", data=survey_df)

ax2 = fig.add_subplot(222)
sns.barplot(x="Ind_JS", y="Graduate_Date", data=survey_df)

plt.tight_layout()
plt.show()
#Plot the remaining three Industry variables 
fig = plt.figure()

ax3 = fig.add_subplot(221)
sns.barplot(x="Ind_Python", y="Graduate_Date", data=survey_df)

ax4 = fig.add_subplot(223)
sns.barplot(x="Ind_SAS", y="Graduate_Date", data=survey_df)

ax5 = fig.add_subplot(224)
sns.barplot(x="Ind_R", y="Graduate_Date", data=survey_df)

plt.tight_layout()
plt.show()
#Learning Python seems to be an important skills for data science students
#Let's take a look at the distribution of the Ind_Python variable 
#The next block of code will output the untransformed version of the variable 

# Select variable that we want to examine and eliminating missing data 
X = survey_df['Ind_Python'].dropna()

unscaled_fig, ax = plt.subplots()
sns.distplot(X).set_title('Unscaled Ind_Python')
unscaled_fig.savefig('Transformation-Unscaled' + '.pdf', 
    bbox_inches = 'tight', dpi=None, facecolor='w', edgecolor='b', 
    orientation='portrait', papertype=None, format=None, 
    transparent=True, pad_inches=0.25, frameon=None)  

plt.show()
#Scale the Ind_Python using scikit-learn's StandardScaler transformer 
X2 = survey_df['Ind_Python'].dropna().values.reshape(-1,1)

standard_fig, ax = plt.subplots()
sns.distplot(StandardScaler().fit_transform(X2)).set_title('StandardScaler')
standard_fig.savefig('Transformation-StandardScaler' + '.pdf', 
    bbox_inches = 'tight', dpi=None, facecolor='w', edgecolor='b', 
    orientation='portrait', papertype=None, format=None, 
    transparent=True, pad_inches=0.25, frameon=None)  

plt.show()<jupyter_output><empty_output><jupyter_text>Key Takeaways* Notice that the shape of the distribution remained the same for the Ind_Python variable after completing a transformation. The values for the Ind_Python variable were the only items that changed. * The colorful seaborn plots show us how different graduating classes feel about the Industry variables * The correlation heat map indicates the sentiment when comparing two software variables at a time* Based on descriptive statistics, learning R and Python are the top two languages that students need to learn for profesional reason   Section 2 - Future Courses Analysis<jupyter_code>#Let's create a new dataframe that includes the data from the student responses
#about the addition of future courses and the number of courses they completed 
future_df = survey_df.loc[:, 'Python_Course_Interest': 'Courses_Completed']
#Partial output the first five lines of new dataframe
future_df.iloc[0:5, 0:3].head()
#This output shows descriptive statistics for future classes variables
print('\nDescriptive statistics for futures_df -----------------')
print(future_df.describe())
#Visualize the future class variable responses based on classes completed 
fig = plt.figure()
ax1 = fig.add_subplot(221)
sns.barplot(x="Courses_Completed", y="Python_Course_Interest", 
            data=survey_df)
ax2 = fig.add_subplot(222)
sns.barplot(x="Courses_Completed", y="Foundations_DE_Course_Interest", 
            data=survey_df)
ax3 = fig.add_subplot(223)
sns.barplot(x="Courses_Completed", y="Analytics_App_Course_Interest", 
            data=survey_df)
ax4 = fig.add_subplot(224)
sns.barplot(x="Courses_Completed", y="Systems_Analysis_Course_Interest", 
            data=survey_df)
plt.tight_layout()
plt.show()
#Let's do a log transformation of the Python_Course_Interest variable
#Log transformation cannot work with 0 values so we will add 1 constant 
#to all observations that had a 0 values
#We feel comfortable doing this because a student with an interest of 0 
#and 1 is immaterial 
X3= survey_df['Python_Course_Interest'].dropna()
X3[X3==0] = 1

log_fig, ax = plt.subplots()
sns.distplot(np.log(X3)).set_title('NaturalLog')
log_fig.savefig('Transformation-NaturalLog' + '.pdf', 
    bbox_inches = 'tight', dpi=None, facecolor='w', edgecolor='b', 
    orientation='portrait', papertype=None, format=None, 
    transparent=True, pad_inches=0.25, frameon=None)  

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0991359177954227, 'fraction_numerical': 0.012961233068659504, 'mean_word_length': 5.2290909090909095, 'pattern_counts': {'":': 2, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '422818', 'license': 'Unlicense', 'n_tokens_mistral': 2795, 'n_tokens_neox': 2574, 'n_words': 849, 'path': 'Assignment1_OscarHernandez.ipynb', 'repo_name': 'oschernan/Practical-Machine-Learning', 'size': '8589'}