<jupyter_start><jupyter_text><NAME> / trtin1  Research question/interestsI'm interested in whether there's a correlation between a developer's educational background and salary or whether their industrial or experience has a greater impact on their salary.<jupyter_code>import numpy as np
import pandas as pd
import project_functions2 as p2
import seaborn as sns
import matplotlib.pylab as plt
df2021 = pd.read_csv('../data/raw/survey_results_public_2021.csv')
df2021.describe()
df2021.tail()<jupyter_output><empty_output><jupyter_text>Milestone 3   **Task1** : EDA<jupyter_code>print(f"Number of rows and Columns: {df2021.shape}") 
print(f"Column names in the dataframe: {df2021.columns}") 
p2.groupby_EdLevel(df2021).describe
df2021.describe(include='object').T<jupyter_output><empty_output><jupyter_text>**Task2-3** : Load and Clean Data<jupyter_code>#Clean Dataset by removing irrelevent columns and N/A values from Edlevel, LearnCOde, ConvertedCompYearly and Employment
df2021Cleaned = p2.load_and_process('../data/raw/survey_results_public_2021.csv') 
df2021Cleaned['YearsCodePro'] = df2021Cleaned['YearsCodePro'].astype(int)
df2021Cleaned.head()
df2021Cleaned.to_csv('../data/processed/df2021E.csv')
df2021Cleaned.describe(include='object').T
df2021Cleaned[0: 10].melt()<jupyter_output><empty_output><jupyter_text>**Task4** : Analysis<jupyter_code>plt.rcParams['figure.figsize']=15,15
sns.set(font_scale = 1.5)
plt.xticks(rotation= 0)
plt.title("Distribution of Educational Level")
plt.tight_layout()
plt1 = sns.countplot(data = df2021Cleaned, x = df2021Cleaned.EdLevel, hue = 'EdLevel', dodge=False)
plt.legend(title = "Educational Level")
plt1.set_xlabel("Educational Level")
plt1.set_ylabel("Count")
plt1.set(xticklabels=[])

p2.show_Percentage(df2021Cleaned, plt1)<jupyter_output><empty_output><jupyter_text>**observations*** Almost half of the employed respondees have a bachelor's degree* 92.4% of the respondees attempted or acquired an undergraduate or graduate degree* 29.1% of the respondees acquired a graduate degree* Only 0.5% of the respondess claim that primary school is their highest education level<jupyter_code>plt.rcParams['figure.figsize']=20,15
sns.set(font_scale = 1.5)
plt.xticks(rotation= 0)
plt.title("Educational Level VS Total Yearly Compensation")
plt2 = sns.barplot(data = df2021Cleaned, y='EdLevel',x='ConvertedCompYearly', hue = 'EdLevel', dodge=False, )
plt2.legend(title = "Educational Level")
plt2.legend(loc='upper right')
plt2.set_xlabel('Average Total Yearly Compensation')
plt2.set(yticklabels=[])
plt2.set_ylabel("Education Level")<jupyter_output><empty_output><jupyter_text>**observations*** The previous graph shows that only 0.5% of the respondess claim that primary school is their highest education level, but in this plot, respondess with an education level of primary school have the highest total average yearly compensation and also the longest error bar.* This indicates that some respondess intentionally put extreme numbers when participating in the survey.<jupyter_code>dfModiofied = df2021Cleaned.drop(df2021Cleaned[(df2021Cleaned.EdLevel == "Primary/elementary school")].index)
plt.rcParams['figure.figsize']=20,15
sns.set(font_scale = 1.5)
plt.xticks(rotation= 0)
plt.title("Educational Level VS Total Yearly Compensation")
plt2 = sns.barplot(data = dfModiofied, y='EdLevel',x='ConvertedCompYearly', hue = 'EdLevel', dodge=False, )
plt2.legend(title = "Education Level")
plt2.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt2.set_xlabel('Average Total Yearly Compensation')
plt2.set(yticklabels=[])
plt2.set_ylabel("Education Level")<jupyter_output><empty_output><jupyter_text>**observations*** In order to conduct further analysis, all rows with EdLevel of primary school have been removed in the dataset.* Generally speaking, respondees with some sort of post secondary back ground generally make more than those who don't* Education level might not be the only factor contributing to how much one makes yearly.<jupyter_code>plt3 = sns.boxplot(data = df2021Cleaned, x = 'EdLevel',y = 'YearsCodePro',hue = 'EdLevel',dodge=False)
plt3.legend(loc='upper right')
plt3.set_xlabel('Education level')
plt3.set(xticklabels=[])
plt3.set_ylabel("Years of professional coding")<jupyter_output><empty_output><jupyter_text>**observations*** Respondees with a doctoral degree also have the highest average years coding professionally. This is somewhat surprising because they spend more years in schools than respondees with an undergraduate degree and potentially have less experience in the professional field.<jupyter_code>dfModiofied = df2021Cleaned.drop(df2021Cleaned[(df2021Cleaned.EdLevel == "Primary/elementary school")].index)
plt4 = sns.boxplot(data = dfModiofied,y='ConvertedCompYearly',x='YearsCodePro', showfliers = False)
plt4.set_xlabel('Years coding professionally')
plt4.set_ylabel("Total yearly compensations")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1107273466070703, 'fraction_numerical': 0.03819585534335636, 'mean_word_length': 7.806797853309481, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '663092', 'license': 'MIT', 'n_tokens_mistral': 1750, 'n_tokens_neox': 1577, 'n_words': 409, 'path': 'notebooks/analysis2.ipynb', 'repo_name': 'data301-2021-winter1/project-group13-project', 'size': '4926'}