<jupyter_start><jupyter_text>**Data Prepration for Salary Analysis for Programmers/Developers** OverviewIn this dataset we are going to clean the varius columns and prepare our data using various methods. In the analysis section we will find relationships to the various fileds and the salary of Programmers/Developers  **DATASET OVERVIEW**  Importing the libraries we need into the colab env<jupyter_code># Importing the pandas library into our environment
import pandas as pd

# import numpy library into our environment
import numpy as np

#importing matplotlib into our environment 
import matplotlib.pyplot as plt
% matplotlib inline 
from google.colab import drive
drive.mount('/content/drive')<jupyter_output>Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).<jupyter_text>Importing the dataset into our environment<jupyter_code>dd=pd.read_csv("survey_results_public.csv")
# Cheking the first 5 entries into our dataset 
dd.head(5)<jupyter_output><empty_output><jupyter_text>Checking information on our dataset<jupyter_code>dd.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 64461 entries, 0 to 64460
Data columns (total 61 columns):
 #   Column                        Non-Null Count  Dtype  
---  ------                        --------------  -----  
 0   Respondent                    64461 non-null  int64  
 1   MainBranch                    64162 non-null  object 
 2   Hobbyist                      64416 non-null  object 
 3   Age                           45446 non-null  float64
 4   Age1stCode                    57900 non-null  object 
 5   CompFreq                      40069 non-null  object 
 6   CompTotal                     34826 non-null  float64
 7   ConvertedComp                 34756 non-null  float64
 8   Country                       64072 non-null  object 
 9   CurrencyDesc                  45472 non-null  object 
 10  CurrencySymbol                45472 non-null  object 
 11  DatabaseDesireNextYear        44070 non-null  object 
 12  DatabaseWorkedWith            49537 non-null  object 
 13  D[...]<jupyter_text>**DATASET PREPRATION**  Dropping Columns that we do not need into our dataset<jupyter_code>#Dropping the columns listed in the drop section because they will not help with our analysis 

dd.drop(['Respondent','CompTotal','CurrencyDesc','CurrencySymbol',
        'DatabaseDesireNextYear','JobSeek','LanguageDesireNextYear','MiscTechDesireNextYear'],axis=1,inplace=True)

#Cheking if the columns have been dropped
dd.head(3)
#Dropping the columns listed in the drop section because they will not help with our analysis 
dd.drop(['MiscTechWorkedWith','NEWCollabToolsDesireNextYear','NEWDevOps','NEWDevOpsImpt','NEWJobHunt',
        'NEWJobHuntResearch','NEWOffTopic','NEWOnboardGood'],axis=1,inplace=True)

#Cheking if the columns have been dropped
dd.head(3)
#Dropping the columns listed in the drop section because they will not help with our analysis 
dd.drop(['NEWPurchaseResearch','NEWPurpleLink',
         'NEWSOSites','NEWStuck','PlatformDesireNextYear','SurveyEase','SurveyLength'],axis=1,inplace=True)

#Cheking if the columns have been dropped
dd.head(3)
#Dropping the columns listed in the drop section because they will not help with our analysis 
dd.drop([ 'SOAccount', 'SOComm','SOPartFreq','SOVisitFreq','Trans','WelcomeChange','WebframeWorkedWith','WebframeDesireNextYear'],axis=1,inplace=True)

#Cheking if the columns have been dropped
dd.head(3)<jupyter_output><empty_output><jupyter_text>**Checking for duplicates in our columns**<jupyter_code># Cheking if our dataset contains any duplicate values
dd.duplicated().values.any()
#Dropping duplicates
dd.drop_duplicates()<jupyter_output><empty_output><jupyter_text>**Cheking for missing values in our columns**<jupyter_code>#Cheking the sum of msissing values in our columns
dd.isnull().sum()<jupyter_output><empty_output><jupyter_text>**Taking care of missing values**<jupyter_code># Dropping the missing values in our convertedcomp column as it conatins the salary in usd for the survey
# also cheking if the missing values were dropped in that colunmn 
dd2= dd.dropna(how='any', subset= ['ConvertedComp'])
dd2.isnull().sum()<jupyter_output><empty_output><jupyter_text>**Imputing the mean for the null values in the integer columns**<jupyter_code># imputing the mean for the null values in the age column 
dd2['Age'].fillna(dd2['Age'].mean(), inplace= True)

# imputing the mean for the null values in the work_week hours column 
dd2['WorkWeekHrs'].fillna(dd2['WorkWeekHrs'].mean(), inplace= True)


#Filling missing values in the Age1stcode column with zero
dd2["Age1stCode"].fillna(0, inplace= True)

#Filling missing values in the yearcode column with zero
dd2["YearsCode"].fillna(0, inplace= True)


#Filling missing values in the yearcodepro column with zero
dd2["YearsCodePro"].fillna(0, inplace= True)


# Cheking the sum of missing values
dd2.isnull().sum()<jupyter_output>/usr/local/lib/python3.7/dist-packages/pandas/core/series.py:4536: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  downcast=downcast,<jupyter_text>**Filling the missing values in the string with a .**<jupyter_code>missing_strl = dd2[["DatabaseWorkedWith", "DevType", "EdLevel", "Employment", "Ethnicity", "Gender", "JobFactors", "JobSat",
                "LanguageWorkedWith", "NEWCollabToolsWorkedWith", "NEWEdImpt", "NEWOtherComms", "NEWOvertime", "OpSys", 
                "OrgSize", "PlatformWorkedWith", "PurchaseWhat", "Sexuality", "UndergradMajor", "NEWLearn"]]

#Creating a for loop to iterate over the column names in missing_strl and where there is a nan value it repalces with a dot instead
for i in missing_strl:
  dd2[i].replace(np.nan, ".", inplace= True)

dd2.head()
#cheking the sum of null values after replacing 
dd2.isnull().sum()<jupyter_output><empty_output><jupyter_text>**Taking care of special characters in the dataset**<jupyter_code># Creating a list of special characters
special_characters = ["™", "€"]

# Creating a for loop to iterate over the names in the columns
# and remove the special characters from the columns if they are found
# existing in the column
for sch in special_characters:
    dd2['EdLevel'] = dd2['EdLevel'].str.replace(sch, ' ')

#Since we replaced the special characters with a plus we might end up with double white spaces 
# we will remove the double white spaces and rejoin the words 
dd2['EdLevel']= dd2['EdLevel'].str.split().str.join(" ")


#Cheking the column to see whether the special charcters listed have been removed
dd2[["EdLevel"]]<jupyter_output>/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:8: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  
/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:12: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  if sys.path[0] == '':<jupyter_text>**Dealing with our Outliers**  Quantile calculations<jupyter_code>Q1 = dd2.quantile(0.25)
Q3 = dd2.quantile(0.75)
IQR = Q3 - Q1<jupyter_output><empty_output><jupyter_text>Cheking the sum of outliers in each column<jupyter_code>((dd2 < (Q1 - 1.5 * IQR)) | (dd2 > (Q3 + 1.5 * IQR))).sum()<jupyter_output><empty_output><jupyter_text>Visualising our outliers in histograms to check skewness<jupyter_code>dd2.hist(column = ['WorkWeekHrs', 'Age', 'ConvertedComp'])<jupyter_output><empty_output><jupyter_text>**Histogram outcome of the dataset**As we can see from the histogram obove our data is skewed right meaning that the position of most of our outliers are present on the right side of the distribution, the visialisation will not tell us the number of outlier but only the direction  Visualising our outliers in box plots to see the outliers<jupyter_code>#checking the number of of outliers from our salary data 
dd2.boxplot(column =['ConvertedComp'], grid=False)
#checking the number of of outliers from our age data 
dd2.boxplot(column =['Age'], grid=False)<jupyter_output><empty_output><jupyter_text>Dropping specific outliers in the dataset that users are 80 years old<jupyter_code>dd2.drop(dd2[dd2['Age']>80].index,inplace=True)

#cheking the shape of our dataset after dropping
dd2.shape
#checking the number of of outliers from our salary data 
dd2.boxplot(column =['WorkWeekHrs'], grid=False)<jupyter_output><empty_output><jupyter_text>Dropping specific outliers in the dataset that users work for 100 hours a week<jupyter_code>dd2.drop(dd2[dd2['WorkWeekHrs']>100].index,inplace=True)

#cheking the shape of our dataset after dropping
dd2.shape<jupyter_output>/usr/local/lib/python3.7/dist-packages/pandas/core/frame.py:4174: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  errors=errors,<jupyter_text>**Outliers Outcome for the dataset**The outliers in the three columns are very significant in-regards to our data, dropping the outliers might resort in missing out on good findings within the dataset, one of the questions we should ask our-selves is are those even outliers especially based on the analysis we are trying to compute and find, so in this case since the outliers are part of our data not dropping them might actually help with our analysisThat being said we can take two separate datasets one with dropped outliers and another without and compare them and see if we can make any conclusions  Exporting our cleaned dataset into csv format<jupyter_code>dd2.to_csv("survey-results-filtered.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09312962242117555, 'fraction_numerical': 0.019657454262358893, 'mean_word_length': 4.780089988751406, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 68, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '408284', 'license': 'MIT', 'n_tokens_mistral': 3341, 'n_tokens_neox': 3120, 'n_words': 1029, 'path': 'Salary_Analysis_for_Developers_Programmers(Lilac_Lotus_group).ipynb', 'repo_name': 'alexgathuri/salary-analysis-for-developers', 'size': '10276'}