<jupyter_start><jupyter_text>Section 1 Business Understanding The purpose of this project is to use Stack Overflow data from 2020 to better understand: Question 1: Which programming languages are the most desired next year (Which language to learn in 2021)? Question 2: Which programming languages are the most popular among developers in 2020? Question 3: How much more do developers earn for mastering certain programming languages?<jupyter_code># importing the relevant libararies
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score<jupyter_output><empty_output><jupyter_text>Secion 2 Data UnderstandingThe main data source is the survey results from Stack Overflow which is an open source and can be access through the link in the Readme file.   Gather<jupyter_code>#importing the data
df = pd.read_csv("survey_results_public.csv")
df.head()<jupyter_output><empty_output><jupyter_text>Assess<jupyter_code>#getting an overview of the numerical data in the dataset 
df.describe()<jupyter_output><empty_output><jupyter_text>Section 3 Data PreparationAs there are columns which are mixed of string and float datatypes, this first needs to be imputed using the imputation function defined above. This enables us to have more quantitative data from the dataset. Furthermore, there are missing values in some of the relevant variables which also need to be preprocessed before we can continue with the analysis.<jupyter_code>def impute_func(col):
    '''
    INPUT:
    col - column name that needs to be imputed. 
    OUTPUT:
    columns values that are the string datatypes are imputed with the edge values in the strings.
    
    Imputes the input columns with the mean of the column values for the relevant variables. As the edge cases in these categories
    are quite small compared to the overrall sample size, the edge numbers are used as imputation numbers. For example, 
    'Less than 1 year' is replaced with nubmer 1.
    '''
    if col == 'YearsCode' or col == 'YearsCodePro':
        df[col] = np.where((df[col] == 'Less than 1 year'),1,df[col])
        df[col] = np.where((df[col] == 'More than 50 years'),50,df[col])
    else:
        df[col] = np.where((df[col] == 'Younger than 5 years'),5,df[col])
        df[col] = np.where((df[col] == 'Older than 85'),85,df[col])<jupyter_output><empty_output><jupyter_text>Clean<jupyter_code>#imputing the YearsCode,Age1stCode,YearsCodePro columns as the values are mixed of float and string data types, the impute_func
#cleans the columns so they can be used for analysis later 
impute_func('YearsCode')
impute_func('Age1stCode')
impute_func('YearsCodePro')<jupyter_output><empty_output><jupyter_text>Analyze<jupyter_code>##dealing with null values and removing rows where there are no values for the compensation columns 
##removing any rows without any compensation value because we do not want to assume the mean for the column as that
##people do not have compensation such as students should not have compensation at all 
df.dropna(axis=0,subset=['ConvertedComp'],how='any',inplace = True)
df = df.astype({"YearsCode": float, "Age1stCode": float, "YearsCodePro": float}).drop('CompTotal',axis=1)
df_numerical = df.select_dtypes(include=['float64']).copy()
df_numerical.isnull().sum()
##Given that the number of the missing values in the dataset is relatively small compared to the overall sample size
##and the rows left are only those who are professionals because we cleaned the data without compensation in the previous cell.  
##It is intuitive to replace the null values in the numerical columns with the mean of the columns 
for col in df_numerical.columns:
    df_numerical[col].fillna(df_numerical[col].mean(),inplace=True)<jupyter_output><empty_output><jupyter_text>Section 4 ModellingThe second part of this project focuses on using a simple linear regression model to evaluate the third question that needs to be addressed which is that how much more can developers earn for mastering certain programming languages. The data will be fitted on the model and used to derive the coefficients to draw the conclusions.<jupyter_code>def get_languages(df):
    '''
    INPUT:
    df - the original dataframe is passed in as input.  
    OUTPUT:
    Outputs all the dummy variables for all the programming languages in the answers.
    
    This function takes in the original dataframe and calculates 1 if a developer konws a particular programming language
    and 0 otherwise for that language. Finally the dataframe with dummy variables is returned as output.

    '''
    df = df.copy()
    df.dropna(axis=0,subset=['LanguageWorkedWith'],how='any',inplace = True)
    language_ranks = df['LanguageWorkedWith'].str.split(';', expand=True).stack().value_counts()
    languages = list(language_ranks.index)
    df['NumberofLanguages'] = df['LanguageWorkedWith'].str.split(';').apply(len)
    for language in languages:
        df[f'Knows_{language}'] = np.where(df['LanguageWorkedWith'].str.find(language) != -1,1,0)
    cols = [col for col in df.columns if 'Knows' in col]  
    cols.extend(['NumberofLanguages','ConvertedComp'])
    return df[cols] 
def coef_weights(coefficients, X_train):
    '''
    INPUT:
    coefficients - the coefficients of the linear model 
    X_train - the training data, so the column names can be used
    OUTPUT:
    coefs_df - a dataframe holding the coefficient, estimate, and abs(estimate)
    
    Provides a dataframe that can be used to understand the most influential coefficients
    in a linear model by providing the coefficient estimates along with the name of the 
    variable attached to the coefficient.
    '''
    coefs_df = pd.DataFrame()
    coefs_df['est_int'] = X_train.columns
    coefs_df['coefs'] = lm_model.coef_
    coefs_df['abs_coefs'] = np.abs(lm_model.coef_)
    coefs_df = coefs_df.sort_values('abs_coefs', ascending=False)
    return coefs_df<jupyter_output><empty_output><jupyter_text>Model<jupyter_code>#Dividing the dataset into test and training sets, fitting the model and finally predict the results
df_final = get_languages(df)
X = df_final.drop('ConvertedComp',axis=1)
y = df_final['ConvertedComp']
lm_model = LinearRegression(normalize=True)
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.3, random_state=42)
lm_model.fit(X_train,y_train)
y_preds = lm_model.predict(X_test)<jupyter_output><empty_output><jupyter_text>Section 5 EvaluationThe graphs for the most populuar languages are plotted using the function defined previously and lastly the cofficients of the variables are ranked based on the size of their absolute values to give an interpretation of their impact on the salary level of the developers.<jupyter_code>def make_graphs(col_name):
    '''
    INPUT:
    col_name - column name that is used to produce the graphs.  
    OUTPUT:
    Produces graphs based on the input column. 
    
    The function creates the horizontal bar chart for the top ranked programming languages based on their popularity among developers
    '''
    language_ranks = df[col_name].str.split(';', expand=True).stack().value_counts()
    df_language_ranks = pd.DataFrame({'Language':language_ranks.index, 'Frequency':language_ranks.values}).sort_values('Frequency')
    ax = df_language_ranks.plot(x='Language',y='Frequency',kind='barh',figsize=(8, 10), color='#86bf91', zorder=2, width=0.85)
    ax.set_xlabel("Popularity", labelpad=20, weight='bold', size=12)
    ax.set_ylabel("Languages", labelpad=20, weight='bold', size=12)
    if col_name == 'LanguageDesireNextYear':
        ax.set_title("Most Desired Programming Languages in 2021", weight='bold', size=14)
    else:
        ax.set_title("Most Popular Programming Languages in 2020", weight='bold', size=14)
    ax.get_legend().remove()
    fig = ax.get_figure()
    fig.savefig(f'{col_name}.png');<jupyter_output><empty_output><jupyter_text>Visualize  Question 1: Which programming languages are the most desired next year (Which language to learn in 2021)?<jupyter_code>#plotting the graph for the most desired programming languages in 2021
make_graphs('LanguageDesireNextYear')<jupyter_output><empty_output><jupyter_text>Question 2: Which programming languages are the most popular among developers in 2020?<jupyter_code>#plotting the graph for the most desired programming languages in 2020
make_graphs('LanguageWorkedWith')<jupyter_output><empty_output><jupyter_text>Question 3: How much more do developers earn for mastering certain programming languages?<jupyter_code>#Use the function to plot top variables based on coefficients
coef_df = coef_weights(lm_model.coef_, X_train)
#A quick look at the top results
coef_df.head(5).reset_index().drop('index',axis=1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08472472924187725, 'fraction_numerical': 0.011845667870036101, 'mean_word_length': 5.229796205200281, 'pattern_counts': {'":': 3, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '514303', 'license': 'CNRI-Python', 'n_tokens_mistral': 2675, 'n_tokens_neox': 2538, 'n_words': 926, 'path': 'Data Science Nanodegree Project.ipynb', 'repo_name': 'forestphilosophy/Data-Science-Nanodegree', 'size': '8864'}