<jupyter_start><jupyter_text>A Data Analysis using Stackoverflow’s 2017 and 2018 Annual Developer Survey.  Each stage of the CRISP-DM process. These stages are:1. Business Understanding2. Data Understanding3. Prepare Data4. Data Modeling5. Evaluate the Results  Business UnderstandingIn this notebook, I will be exploring the 2017 and 2018 Stack Overflow results in order to glean some information on professional developers and what to focus on to have the best career in India.There will be three questions I will seek to answer in order to get this information:1. What Programming Languages are most used to work and Which Programming Languages are most Required in Stackoverflow survey data of 2017 and 2018 ?2. How does Programming Languages used at work relates with Programming Languages, People Wants to Learn in India According to Stackoverflow survey data of 2017 and 2018?3. What are the most wanted Programming Languages in India According to Stackoverflow survey data of 2017 and 2018?  Data UnderstandingIn order to gain some understanding of the data. I have to do these steps:* Handle categorical and missing data* Analyze, Model, and Visualize<jupyter_code># import python libraries to handle datasets
import numpy as np
import pandas as pd
from collections import Counter

#make plots
import matplotlib.pyplot as plt
import seaborn as sns

#ignore warnings
import warnings
warnings.filterwarnings("ignore")

# to display graphs in jupyter notebook
%matplotlib inline

# to visualise al the columns in the dataframe
pd.pandas.set_option('display.max_columns', None)

from jupyterthemes import jtplot
jtplot.style(theme='onedork')
# Loading Stackoverflow survey data of 2017 and 2018
# Ref link: https://insights.stackoverflow.com/survey
df_2017 = pd.read_csv('2017/survey_results_public.csv')
df_2018 = pd.read_csv('2018/survey_results_public.csv')
#Little Exploration on Stackoverflow survey data of 2017
df_2017.sample(5)
df_2017.head(3)
#Little Exploration on Stackoverflow survey data of 2018
df_2018.sample(5)
df_2018.head(3)<jupyter_output><empty_output><jupyter_text>Question 1: What Programming Languages are most used to work and Which Programming Languages are most Required in  Stackoverflow survey data of 2017 and 2018 ?  Prepare Data<jupyter_code>#First make copy of dataframe and then Filter a dataframe by country 
def filter_country(df, column_filter, country, columns): 
    '''
    Filter a dataframe by country(i.e. why i use dropna to drop all other country excpet India)
    Returns filtred dataframe
    '''
    
    df_copy = df
    
    for column in columns: 
        df_copy = df_copy[df_copy[column_filter] == country].dropna(subset=[column])
        
    return df_copy

# Filtering the dataframe 
india_2017 = filter_country(df_2017, 'Country', 'India', ['HaveWorkedLanguage', 'WantWorkLanguage'])
india_2018 = filter_country(df_2018, 'Country', 'India', ['LanguageWorkedWith', 'LanguageDesireNextYear'])
#Spliting the columns
def split_column(df, column):
    '''
    Split column by ;,
    Returns a splited series.
    '''
    df_copy = df
    columnSeries = df_copy[column].apply(lambda x: x.split(';'))
    return columnSeries

# Splitting the dataframe by columns.
worked_languages_2017 = split_column(india_2017, 'HaveWorkedLanguage')
wanted_languages_2017 = split_column(india_2017, 'WantWorkLanguage')

worked_languages_2018 = split_column(india_2018, 'LanguageWorkedWith')
wanted_languages_2018 = split_column(india_2018, 'LanguageDesireNextYear')
#Just Flating a nested list
def flat(array_list):
    '''
    Flat a nested list,
    Returns a flat list.
    '''
    object_list = []
    for row in array_list:
        for obj in row:
            object_list.append(obj.strip())
            
    return object_list

# Flatting nested list objects.
list_worked_languages_2017 = flat(worked_languages_2017)
list_wanted_languages_2017 = flat(wanted_languages_2017)

list_worked_languages_2018 = flat(worked_languages_2018)
list_wanted_languages_2018 = flat(wanted_languages_2018)
def list_of_group(data_list, year):
    '''
    Group by count to a list,
    Returns a result dict
    '''
    grouped_list = dict(Counter(data_list))
    grouped_dict = [{'Programming Language':key, 'Count': value, 'Year': year} for key, value in grouped_list.items()]
    return grouped_dict

# Grouping the list and creating a dict.
dict_worked_languages_2017 = list_of_group(list_worked_languages_2017, '2017')
dict_wanted_languages_2017 = list_of_group(list_wanted_languages_2017, '2017')

dict_worked_languages_2018 = list_of_group(list_worked_languages_2018, '2018')
dict_wanted_languages_2018 = list_of_group(list_wanted_languages_2018, '2018')<jupyter_output><empty_output><jupyter_text>Data Modelling<jupyter_code>#Ref: https://stackoverflow.com/questions/23668427/pandas-three-way-joining-multiple-dataframes-on-columns
def create_dataframe(data_dicts): 
    '''
    Create two dataframes and append them,
    Returns a appended dataframe.
    '''
    df1 = pd.DataFrame(data_dicts[0])
    df2 = pd.DataFrame(data_dicts[1])
    df = df1.append(df2)
    return df

worked_languages = create_dataframe([dict_worked_languages_2017, dict_worked_languages_2018])
wanted_languages = create_dataframe([dict_wanted_languages_2017, dict_wanted_languages_2018])
#Adding Percentage to worked_languages and wanted_languages dataframe
def percentage(df, column):
    '''
    Scale data,
    Returns data scaled.
    '''
    df_copy = df
    series = []
    for val in df_copy[column].unique():
        series.append(df_copy[df_copy[column] == val]['Count'] / 
                        df_copy[df_copy[column] == val]['Count'].sum())
    
    joined = pd.Series()
    for i_series in series:
        joined = joined.append(i_series)
        
    return joined

worked_languages['Percentage'] = percentage(worked_languages, 'Year')
wanted_languages['Percentage'] = percentage(wanted_languages, 'Year')
# Get the top 12 languages
top_12_work = worked_languages.sort_values(by=['Percentage'], ascending=False).head(12)['Programming Language'].unique()
top_12_want = wanted_languages.sort_values(by=['Percentage'], ascending=False).head(12)['Programming Language'].unique()

worked_chart = worked_languages[worked_languages['Programming Language'].isin(top_12_work)]
wanted_chart = wanted_languages[wanted_languages['Programming Language'].isin(top_12_want)]
plt.figure(figsize=(20,10))
sns.barplot(x = 'Programming Language', 
            y = 'Percentage',
            hue = 'Year',
            data = worked_chart.sort_values(by='Percentage', ascending=False))

plt.xlabel("Programming Languages", fontsize = 14)
plt.ylabel("Percentage", fontsize = 14)
plt.legend(title_fontsize='40')
plt.title('Most common Programming Languages used in India', size = 16)
plt.show()<jupyter_output><empty_output><jupyter_text>Evaluate the Results* The Highest Rate of percentage among all Programming Languages JavaScript have highest growth rate in 2017 which is around 16% whereas this percentage is drop significantly in 2018 about 4% in India.* As we can see through Graph, Java has same drop percentage about 4% in India.* The Most Intersting this about this Graph is `HTML and CSS` it was probably not possible to select these two Languages as option in 2017 whereas in 2018 these two Programming Languages have risen such as `HTML and CSS`.  Question 2: How does Programming Languages used at work relates with Programming Languages, People Wants to Learn in India According to Stackoverflow survey data of 2017 and 2018?  Prepare Data<jupyter_code>row = []
for j in list(zip(worked_languages_2018, wanted_languages_2018)):
    for i in j[0]:
        row.append({
            'Worked_Programming_Languages': i, 
            'Wanted_Programming_Languages': Counter(j[0])
        })
        
programming_language_transition = pd.DataFrame(row).groupby('Worked_Programming_Languages')\
                                            .agg({'Wanted_Programming_Languages': 'sum'}).reset_index()<jupyter_output><empty_output><jupyter_text>Data Modeling<jupyter_code>for row in programming_language_transition['Worked_Programming_Languages']:
    programming_language_transition[row] = 0

for index, row in programming_language_transition.iterrows():
    total = sum([value for key, value in dict(row['Wanted_Programming_Languages']).items()])
    for key, value in dict(row['Wanted_Programming_Languages']).items():
         programming_language_transition[key].loc[index] = (value / total)
prlt = programming_language_transition.drop('Wanted_Programming_Languages', axis=1)\
                                        .set_index('Worked_Programming_Languages')

plt.figure(figsize=(20, 10))
sns.heatmap(prlt, cmap = "Blues")
plt.title('Programming Languages Heatmap', size = 20)
plt.ylabel('')
plt.show()<jupyter_output><empty_output><jupyter_text>Evaluate the Results* With this Graph we can have some insights and they are:  * `HTML` is highly correlated with every Programming Language (except: Rust, Ocaml, Julia, Hack, Haskell and Clojure) and `CSS` also have same trends like HTML.  * There are around 27 Programming Language which has nearly no correlation among them with anyone.  * `HTML` has strongest correlation between `CSS` and `JavaScript` (i.e. about `12% to 14%`).  * There is one noticeable thing that `Objectice C` and `Swift` are strongly correlated with each other (i.e. about `10% to 12%`).  Question 3: What are the most wanted Programming Languages in India According to Stackoverflow survey data of 2017 and 2018?<jupyter_code>plt.figure(figsize=(16,10))
sns.barplot(x = 'Programming Language', 
            y = 'Percentage',
            hue = 'Year',
            data = worked_chart.sort_values(by='Percentage', ascending=False))

plt.title('Most wanted Programming Languages used in India', size = 16)
plt.xlabel("Programming Languages", fontsize = 12)
plt.ylabel("Percentage", fontsize = 12)
plt.legend(title_fontsize='40')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08898220355928814, 'fraction_numerical': 0.03709258148370326, 'mean_word_length': 4.894519740718915, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '437338', 'license': 'MIT', 'n_tokens_mistral': 3304, 'n_tokens_neox': 2887, 'n_words': 908, 'path': 'Project - Stack Overflow Developer Survey.ipynb', 'repo_name': 'Narendrasai-Bathula/write-datascience_blog-', 'size': '10002'}