<jupyter_start><jupyter_text>By <NAME>In this notebook, we are going to take a look at the stack overflow survey 2019 and I will focus mainly on french developers data. I will try to answer 3 main business questions in the notebook:____  1. What technologies do they use and how are the technologies are asscociated with the salaries?  2. What are the developer types and the breakdown salaries that come with?  3. An investigation on the education background of french developers?<jupyter_code>import pandas as pd
from collections import defaultdict
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import textwrap as tw
data = pd.read_csv('survey_results_public.csv')
# Take a look at all 85 columns of the dataframe
with pd.option_context('display.max_rows', 5, 'display.max_columns', None): 
    display(data)
#Select only France data
Francedata= data.query("Country=='France'")
Francedata.describe()
#I will mainly focus on the professional developers
data.MainBranch.unique()
Fdf = pd.DataFrame(Francedata.query("MainBranch == 'I am a developer by profession'"))
Fdf.head()
#Exclure 0 values,Here I choose values in USD as I am going to write on a US based media
Fdf = Fdf[Fdf['ConvertedComp'] != 0]

print(Fdf.ConvertedComp.isnull().sum())

Fdf.describe()<jupyter_output><empty_output><jupyter_text>Part I Check the technologies and developer types___ Here we use the StackOverflow data to attempt to answer our questions of interest. With regard to the data science CRISP-DM Process, the business questions and data questions are kind of mixed. Now we are going to prepare the data to answer the questions.____As the codes for the findings are similar, I will work with them one after another<jupyter_code>Fdf.LanguageWorkedWith.unique()<jupyter_output><empty_output><jupyter_text>We can see above that we need to clean this column so as to extract intersting foundings. I am going to count each of the language and find the percentage with regard to the number of respondents.<jupyter_code># Prepare the cleaning of the dataframe
def total_count(df, col1, col2, look_for):
    '''
    INPUT:
    df - the pandas dataframe we want to search
    col1 - the column name we want to look through
    col2 - the column we want to count values from
    look_for - a list of strings we want to search for in each row of df[col]

    OUTPUT:
    new_df - a dataframe of each look_for with the count of how often it shows up
    '''
    new_df = defaultdict(int)
    #loop through list of language type
    for val in look_for:
        #loop through rows
        for idx in range(df.shape[0]):
            #if the language type is in the row add 1
            if val in df[col1][idx]:
                new_df[val] += int(df[col2][idx])
    new_df = pd.DataFrame(pd.Series(new_df)).reset_index()
    new_df.columns = [col1, col2]
    new_df.sort_values('count', ascending=False, inplace=True)
    return new_df
#Check the number of respondants to see if the number is significant, this figure is used in next cell
respondant = Fdf['LanguageWorkedWith'].count()
respondant
possible_vals = ["Assembly", "Bash/Shell/PowerShell", 
                 "C", "C++","C#", 
                 "Clojure", "Dart", "Elixir",
                 "Erlang", "F#", "Go",
                 "HTML/CSS", "Java",
                 "JavaScript", "Kotlin", "Objective-C","PHP", "Python", "R",
                 "Ruby", "Rust", "Scala",
                 "SQL", "Swift",
                 "TypeScript", "VBA", "WebAssembly","Other"]

def clean_and_plot(Fdf, title='Languages used for work', plot=True):
    '''
    INPUT 
        df - a dataframe holding the LanguageWorkedWith column
        title - string the title of ourr plot
        axis - axis object
        plot - bool providing whether or not we want a plot back
        
    OUTPUT
        language_df - a dataframe with the count of how many languages
        Displays a plot of pretty things related to the LanguageWorkedWith column.
    '''
    language = Fdf['LanguageWorkedWith'].value_counts().reset_index()
    respondant = Fdf['LanguageWorkedWith'].count()
    language.rename(columns={'index': 'Language', 'LanguageWorkedWith': 'count'}, inplace=True)
    language_df = total_count(language, 'Language', 'count', possible_vals)

    language_df.set_index('Language', inplace=True)
    if plot:
        (language_df/respondant).plot(kind='bar', legend=None,figsize=[8,6]);
        y_ticks = np.round(np.arange(0,0.85,0.05),2);
        y_ticklabels = pd.Series(y_ticks * 100).round(2).astype(str) + " %";
        plt.yticks(y_ticks,y_ticklabels);
        plt.title(title);
#Add footnote
        comment2_txt = '''1748 responses,select all that apply  
    ''';
        fig_txt = tw.fill(tw.dedent(comment2_txt.rstrip() ), width=60);

# The YAxis value is -0.07 to push the text down slightly
        plt.figtext(0.5, -0.07, fig_txt, horizontalalignment='center',
            fontsize=8, multialignment='left',
            bbox=dict(boxstyle="round", facecolor='#D8D8D8',
                      ec="0.1", pad=0.1, alpha=0.3), fontweight='bold');
    #Save images for my data visualization article
        plt.savefig('Languages_used_for_work.png',bbox_inches='tight')
    props_language_df = language_df/respondant
    print(respondant)
    
    
    return props_language_df
    
props_df = clean_and_plot(Fdf)


# Get the list of the developer type, which I will use next
df_devtype=data.DevType.unique()

df_devtype=df_devtype.tolist()
strings=[]
for i in df_devtype:
    for f in str(i).split(";"):
        strings.append(f)
strings=set(strings)
strings

possible_vals = ['Academic researcher',
 'Data or business analyst',
 'Data scientist or machine learning specialist',
 'Database administrator',
 'Designer',
 'DevOps specialist',
 'Developer, QA or test',
 'Developer, back-end',
 'Developer, desktop or enterprise applications',
 'Developer, embedded applications or devices',
 'Developer, front-end',
 'Developer, full-stack',
 'Developer, game or graphics',
 'Developer, mobile',
 'Educator',
 'Engineer, data',
 'Engineer, site reliability',
 'Engineering manager',
 'Marketing or sales professional',
 'Product manager',
 'Scientist',
 'Senior executive/VP',
 'Student',
 'System administrator',
 'nan']

def clean_and_plot(Fdf, title='Percentage of Developer Types', plot=True):
    '''
    INPUT 
        df - a dataframe holding the DevType column
        title - string the title of ourr plot
        axis - axis object
        plot - bool providing whether or not we want a plot back
        
    OUTPUT
        Devtype_df - a dataframe with the count of how many Developer types
        Displays a plot of pretty things related to the DevType column.
    '''
    Devtype = Fdf['DevType'].value_counts().reset_index()
    Devtype.rename(columns={'index': 'Developer type', 'DevType': 'count'}, inplace=True)
    Devtype_df = total_count(Devtype, 'Developer type', 'count', possible_vals)
    respondant = Fdf['DevType'].count()

    Devtype_df.set_index('Developer type', inplace=True)
    if plot:
        (Devtype_df/respondant).plot(kind='bar', legend=None,figsize=[8,6]);
        plt.title(title);
        y_ticks = np.round(np.arange(0,0.65,0.05),2);
        y_ticklabels = pd.Series(y_ticks * 100).round(2).astype(str) + " %"
        plt.yticks(y_ticks,y_ticklabels);
        #Save images for my data visualization article
        plt.savefig('Percentage of Developer Types.png',bbox_inches='tight')
    props_Devtype_df = Devtype_df/respondant
    return props_Devtype_df
    
props_df = clean_and_plot(Fdf)
# Check company size to see if there is anything interesting
Orgsize = Fdf.groupby('OrgSize').median()['ConvertedComp']
Orgsize<jupyter_output><empty_output><jupyter_text>Part 2 Check the salaries associated with the technologies and developer types_____  data science CRISP-DM ProcessWe did our first data wranglings to extract useful information with regard to the technology and developer type questions. Now we continue to wrangle and extract statitics to answer our questions<jupyter_code>DevType = []
for dev in Fdf.DevType:
    try:
        DevType.extend(dev.split(';'))
    except AttributeError:
        pass

# we end up needing this later
DevType = set(DevType)
print("The number of DevType is {}.".format(len(DevType)))
def split_DevType(val):
    try:
        if val.find(dev) >-1:
            return 1
        else:
            return 0
    except AttributeError:
        return 0

# Apply function for each developer type
for dev in DevType:        
    Fdf[dev] = Fdf['DevType'].apply(split_DevType)
#check if data related with this row is representative, for me, it is ok
Fdf[Fdf['Senior executive/VP']==1]['ConvertedComp'].tolist()
#Make a barplot with 2 variables, developer types and the salaries
Salary = dict()
for dev in DevType:
    Sal_median = Fdf[Fdf[dev]==1].ConvertedComp.median()
    Salary[dev]=Sal_median
Salary = pd.DataFrame(Salary.items(),columns=['Developper type','Salary(USD)'])
Salary = Salary.sort_values(by='Salary(USD)',ascending=False).reset_index(drop=True)
plt.figure(figsize=[10,8])
plt.title("Salary Breakdown by Developer Type")
ax = sns.barplot(y="Developper type", x='Salary(USD)', data=Salary,palette='viridis')
#Add salary amount annotation for each catagorical type
for p in ax.patches:
    width = p.get_width()
    plt.text(5+p.get_width(), p.get_y()+0.65*p.get_height(),
             '{:1.0f}'.format(width),
             ha='left', va='center')
    
plt.savefig('Slary Breakdown by Developer Type',bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>Result 1When it comes to the annual salary(with exchange rate eur vs. usd 0.87), the France follows the same fashion as in the world. The Engineering manager and site reliability engineer jobs are the most rewarded numerically.While the mobile developer is almost the only job that is better paid in France than around the world according to the survey, with 47K vs 45K USD. On the other hand, for an engineering manager, the revenue gap with the world median level can go up to 28K USD annually. If we compare the figures with the same figures in the US, engineering managers earn 55% less than their American fellows and that is 85K less in a year.Wow, that is a huge difference! Luckily the healthcare system and public education systems are all for free, otherwise we cannot imagine how those middle and high level talents in France will flee away.<jupyter_code>LanguageWorkedWith = []
for language in Fdf.LanguageWorkedWith:
    try:
        LanguageWorkedWith.extend(language.split(';'))
    except AttributeError:
        pass

# we end up needing this later
LanguageWorkedWith = set(LanguageWorkedWith)

def split_LanguageWorkedWith(val):
    try:
        if val.find(language) >-1:
            return 1
        else:
            return 0
    except AttributeError:
        return 0

# Apply function for each language
for language in LanguageWorkedWith:        
    Fdf[language] = Fdf['LanguageWorkedWith'].apply(split_LanguageWorkedWith)
    
Salary_language = dict()
for language in LanguageWorkedWith:
    Salary_language_median = Fdf[Fdf[language]==1].ConvertedComp.median()
    Salary_language[language]=Salary_language_median
Salary_language = pd.DataFrame(Salary_language.items(),columns=['LanguageWorkedWith','Salary(USD)'])
Salary_language = Salary_language.sort_values(by='Salary(USD)',ascending=False).reset_index(drop=True)
plt.figure(figsize=[10,8])
ax = sns.barplot(y="LanguageWorkedWith", x='Salary(USD)', data=Salary_language,palette='viridis')
plt.ylabel("Languages Worked With")
plt.title("Salary By Language ")
# Add annotations to each bar
for p in ax.patches:
    width = p.get_width()
    plt.text(5+p.get_width(), p.get_y()+0.65*p.get_height(),
             '{:1.0f}'.format(width),
             ha='left', va='center')
plt.savefig('Salary by Language',bbox_inches='tight')<jupyter_output><empty_output><jupyter_text>Result 2:When we dig more into the languages we find that the median salaries are almost inverse of the language popularity. Clojure,Erlang and F are the most paying languages while among the least mastered 5 languages.On the other hand, the most used languages, as mentioned before, do not seem to generate excellent salaries, they are at the lower half of the ladder.  Part 3 Interesting Studies about french developers' education background---- Use 2 catagorical variables,the undergraduate major and education level,  I choose heatmap as the data viz tool<jupyter_code>Edlevel = pd.DataFrame(Fdf.groupby(['UndergradMajor','EdLevel',]).Respondent.count().sort_values(ascending=False))
Edlevel.reset_index()
Edlevel.Respondent.sum()
#Prepare the table for heatmap, replace the "Nan" values by 0 as the count is 0 for those cases
cat_count = Edlevel.pivot_table(index="UndergradMajor",columns="EdLevel",values="Respondent")
cat_count.fillna(0,inplace=True)
cat_count<jupyter_output><empty_output><jupyter_text>There are more missing values in 'Associate degree' column but I am going to leave it as the non-zero values still interest us. The diversity of the figures is a point that we need to take note.<jupyter_code>sns.heatmap(cat_count, annot = True, fmt = '.0f',
           cbar_kws = {'label' : 'number of counted respondants'},cmap="CMRmap_r")
plt.xticks(rotation=45,ha='right')
plt.xlabel("Education Level")
plt.ylabel("Undergraduate Studies Major")
plt.title("French Developers' Education Background")
plt.savefig('French Developers Education Background',bbox_inches='tight')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10235060054528038, 'fraction_numerical': 0.009726622945987768, 'mean_word_length': 4.0453531598513015, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '197990', 'license': 'CNRI-Python', 'n_tokens_mistral': 4195, 'n_tokens_neox': 4002, 'n_words': 1354, 'path': 'How Much Do French Developers Earn a Year?.ipynb', 'repo_name': 'Youfeiv/Stack-Overflow-Survey-2019', 'size': '13573'}