<jupyter_start><jupyter_text>What type of programmer has the highest level of salary.In order to know what kind of programmer has the highest level of salary. We need to extract the field of salary and developer types. Developer types could be found using the following code:<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_squared_error
from collections import defaultdict
import seaborn as sns
%matplotlib inline

df = pd.read_csv('./survey_results_public.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Data understandingWe will look into the summary of different developer types to get a feeling about what type we will deal with.<jupyter_code>developer_type = df['DeveloperType'].value_counts().reset_index()

developer_type.head()<jupyter_output><empty_output><jupyter_text>Data preparationDeveloperType contains semicolon separated types. So in terms of extracting type, we need to build functions that put all possible types into a set. Also, for removing null values. As we are finding the relationship between salary and developer type, if the row doesn't contain the value we should remove the null row.<jupyter_code># Clean the data
subdf = df[['DeveloperType', 'Salary']]
subdf_dna = subdf.dropna().reset_index()
subdf_dna.head()
def get_possible_types(df, typecol):
    '''
    INPUT:
    df - the pandas dataframe
    typecol - type column name
    
    OUTPUT:
    possible_types - a set of possible developer types
    '''
    possible_types = set()
    developer_type = df[typecol].value_counts().reset_index()
    for typestr in developer_type['index']:
        types = typestr.split(";")
        for devtype in types:
            possible_types.add(devtype.strip())
        
    return possible_types

def to_dict(df, col1, col2, possible_types):
    '''
    INPUT:
    df - the pandas dataframe
    col1 - column to be analysed
    col2 - column contains the value
    possible_types - possible values for col1
    '''
    list_dict = {}
    for val in possible_types:
        list_dict[val] = []
    #loop through list of ed types
    for val in possible_types:
        #loop through rows
        for idx in range(df.shape[0]):
            #if the ed type is in the row add 1
            if val in df[col1][idx]:
                #new_df[val] += int(df[col2][idx])
                list_dict[val].append(df[col2][idx])
    return list_dict
    
def median_col(df, col1, col2, look_for):
    '''
    INPUT:
    df - the pandas dataframe you want to search
    col1 - the column name you want to look through
    col2 - the column you want to get median from
    look_for - a list of strings you want to search for in each row of df[col]

    OUTPUT:
    new_df - a dataframe of each look_for with the count of how often it shows up
    '''
    salarylist_dict = to_dict(df, col1, col2, look_for)
    for key in salarylist_dict.keys():
        sallist = pd.Series(salarylist_dict[key])
        qtiles = [sallist.quantile(0.25), sallist.quantile(0.5), sallist.quantile(0.75)]
        salarylist_dict[key] = qtiles
    
    new_df = pd.DataFrame.from_dict(salarylist_dict)
    new_df = new_df.transpose()
    new_df.columns = ['.25', 'median', '.75']
    new_df.sort_values('median', ascending=False, inplace=True)
    return new_df
    
possible_types = get_possible_types(df, 'DeveloperType')

# Get the quantile points
qpoints = median_col(subdf_dna, 'DeveloperType', 'Salary', possible_types)
qpoints<jupyter_output><empty_output><jupyter_text>EvaluationThe quantile points looks fine, we can now plot it.<jupyter_code>salary_dict = to_dict(subdf_dna, 'DeveloperType', 'Salary', possible_types)

plt.boxplot(salary_dict.values(), showfliers = False, labels = salary_dict.keys(), vert = False)<jupyter_output><empty_output><jupyter_text>What type of job is the most popular one?I'm also interested in what type of developer job is the most popular one. For this step, we can reuse the previous data frame, but this time data cleaning method need to be changed.<jupyter_code>subdf_dnadt = subdf.dropna(subset=['DeveloperType']).reset_index()
type_dict = to_dict(subdf_dnadt, 'DeveloperType', 'Salary', possible_types)

def converttocount(type_dict):
    anotherdict = {}
    for key in type_dict.keys():
        anotherdict.update({key: len(type_dict[key])})
    return anotherdict

totalcount = subdf_dnadt.shape[0]

count_dict = converttocount(type_dict)
type_count = pd.DataFrame(count_dict, index = ['count'])
type_count = type_count.transpose().sort_values(by = 'count', ascending = False)


(type_count/totalcount).plot(kind="bar");
plt.title("What type of job is the most popular one?");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09141675284384695, 'fraction_numerical': 0.0062047569803516025, 'mean_word_length': 4.074501573976915, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '610511', 'license': 'MIT', 'n_tokens_mistral': 1535, 'n_tokens_neox': 1486, 'n_words': 476, 'path': 'DevTypeAndSalary.ipynb', 'repo_name': 'yh214/DST2Proj1', 'size': '4835'}