<jupyter_start><jupyter_text>Exploratory Data Analysis using Python - A Case Study*Analyzing responses from the Stack Overflow Annual Developer Survey 2020* Part 9 of "Data Analysis with Python: Zero to Pandas"This tutorial is the ninth in a series on the introduction to programming and data analysis using the Python language. These tutorials take a practical coding-based approach, and the best way to learn the material is to execute the code and experiment with the examples. Check out the full series here: 1. [First Steps with Python and Jupyter](https://jovian.ml/aakashns/first-steps-with-python)2. [A Quick Tour of Variables and Data Types](https://jovian.ml/aakashns/python-variables-and-data-types)3. [Branching using Conditional Statements and Loops](https://jovian.ml/aakashns/python-branching-and-loops)4. [Writing Reusable Code Using Functions](https://jovian.ml/aakashns/python-functions-and-scope)5. [Reading from and Writing to Files](https://jovian.ml/aakashns/python-os-and-filesystem)6. [Numerical Computing with Python and Numpy](https://jovian.ml/aakashns/python-numerical-computing-with-numpy)7. [Analyzing Tabular Data using Pandas](https://jovian.ml/aakashns/python-pandas-data-analysis)8. [Data Visulation using Matplotlib & Seaborn](https://jovian.ml/aakashns/python-matplotlib-data-visualization)9. [Exploratory Data Analysis - A Case Study](https://jovian.ml/aakashns/python-eda-stackoverflow-survey)  How to run the codeThis tutorial hosted on [Jovian.ml](https://www.jovian.ml), a platform for sharing data science projects online. You can "run" this tutorial and experiment with the code examples in a couple of ways: *using free online resources* (recommended) or *on your own computer*.>  This tutorial is a [Jupyter notebook](https://jupyter.org) - a document made of "cells", which can contain explanations in text or code written in Python. Code cells can be executed and their outputs e.g. numbers, messages, graphs, tables, files etc. can be viewed within the notebook, which makes it a really powerful platform for experimentation and analysis. Don't afraid to experiment with the code & break things - you'll learn a lot by encoutering and fixing errors. You can use the "Kernel > Restart & Clear Output" menu option to clear all outputs and start again from the top of the notebook. Option 1: Running using free online resources (1-click, recommended)The easiest way to start executing this notebook is to click the "Run" button at the top of this page, and select "Run on Binder". This will run the notebook on [mybinder.org](https://mybinder.org), a free online service for running Jupyter notebooks. You can also select "Run on Colab" or "Run on Kaggle", but you'll need to create an account on [Google Colab](https://colab.research.google.com) or [Kaggle](https://kaggle.com) to use these platforms. Option 2: Running on your computer locallyYou'll need to install Python and download this notebook on your computer to run in locally. We recommend using the [Conda](https://docs.conda.io/en/latest/) distribution of Python. Here's what you need to do to get started:1. Install Conda by [following these instructions](https://conda.io/projects/conda/en/latest/user-guide/install/index.html). Make sure to add Conda binaries to your system `PATH` to be able to run the `conda` command line tool from your Mac/Linux terminal or Windows command prompt. 2. Create and activate a [Conda virtual environment](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html) called `zerotopandas` which you can use for this tutorial series:```conda create -n zerotopandas -y python=3.8 conda activate zerotopandas```You'll need to create the environment only once, but you'll have to activate it every time want to run the notebook. When the environment is activated, you should be able to see a prefix `(zerotopandas)` within your terminal or command prompt.3. Install the required Python libraries within the environment by the running the following command on your  terminal or command prompt:```pip install jovian jupyter numpy pandas matplotlib seaborn opendatasets --upgrade```4. Download the notebook for this tutorial using the `jovian clone` command:```jovian clone aakashns/python-eda-stackoverflow-survey```The notebook is downloaded to the directory `python-eda-stackoverflow-survey`.5. Enter the project directory and start the Jupyter notebook:```cd python-eda-stackoverflow-surveyjupyter notebook```6. You can now access Jupyter's web interface by clicking the link that shows up on the terminal or by visiting http://localhost:8888 on your browser. Click on the notebook `python-eda-stackoverflow-survey.ipynb` to open it and run the code. If you want to type out the code yourself, you can also create a new notebook using the "New" button.  IntroductionWe'll use the StackOverflow developer survey dataset for our analysis. This is an annual survey conducted by StackOverflow, and you can find the raw data & results here: https://insights.stackoverflow.com/survey.There are several options for getting the dataset into Jupyter:- Download the CSV manually and upload it via Jupyter's GUI- Use the `urlretrieve` function from the `urllib.request` to download CSV files from a raw URL directly- Use a helper library e.g. [`opendatasets`](https://github.com/JovianML/opendatasets), which contains a collection of curated datasets and provides a function for directly download.We'll use the `opendatasets` helper library to download the files.<jupyter_code>!pip install opendatasets
import opendatasets as od
od.download('stackoverflow-developer-survey-2020')<jupyter_output><empty_output><jupyter_text>Let's verify that the dataset was downloaded into the directory `stackoverflow-developer-survey-2020`, and retrieve the list of files in the dataset.<jupyter_code>import os
os.listdir('stackoverflow-developer-survey-2020')<jupyter_output><empty_output><jupyter_text>You can through the downloaded files using `File > Open` menu option in Jupyter. It seems like the dataset contains 3 files:- `README.txt` - containing information about the dataset- `survey_results_schema.csv` - containing the list of questions (and short codes for each question)- `survey_results_public.csv` - the full list of responses to the questions Let's load the CSV files using the Pandas library. We'll use the name `survey_raw_df` for the data frame, to indicate that this is unprocessed data that which we might clean, filter and modify to prepare a data frame that's ready for analysis.<jupyter_code>import pandas as pd
survey_raw_df = pd.read_csv('stackoverflow-developer-survey-2020/survey_results_public.csv')
survey_raw_df<jupyter_output><empty_output><jupyter_text>The dataset contains over 64,000 responses to 60 questions (although many questions are optional). The responses have been anonymized and there's no personally identifiable information available to us - although each respondent has been assigned a randomized respondent ID.Let's view the list of columns in the data frame.<jupyter_code>survey_raw_df.columns<jupyter_output><empty_output><jupyter_text>It appears that short codes for questions are used as column names. We can refer to the schema file to see the full text of each question. The schema file contains only two columns: `Column` and `QuestionText`, so we can load it as Pandas Series with `Column` as the index and the  `QuestionText` as the value.<jupyter_code>schema_fname = 'stackoverflow-developer-survey-2020/survey_results_schema.csv'
schema_raw = pd.read_csv(schema_fname, index_col='Column').QuestionText
schema_raw<jupyter_output><empty_output><jupyter_text>We can now use `schema_raw` to retrieve the full question text for any column in `survey_raw_df`.<jupyter_code>schema_raw['YearsCodePro']<jupyter_output><empty_output><jupyter_text>We've now loaded the dataset, and we're ready to move on to the next step of preprocessing & cleaning the data for our analysis.  Save and upload your workWhether you're running this Jupyter notebook on an online service like Binder or on your local machine, it's important to save your work from time, so that you can access it later, or share it online. You can upload this notebook to your Jovian.ml account using the `jovian` Python library.<jupyter_code># Select a project name
project='python-eda-stackoverflow-survey'
# Install the Jovian library
!pip install jovian --upgrade --quiet
import jovian
jovian.commit(project=project)<jupyter_output><empty_output><jupyter_text>`jovian.commit` uploads the notebook to your [Jovian.ml](https://jovian.ml) account, captures the Python environment and creates a shareable link for your notebook as shown above. You can use this link to share your work and let anyone (including you) run your notebooks and reproduce your work.  Data Preparation & CleaningWhile the survey responses contain a wealth of information, we'll limit our analysis to the following areas:- Demographics of the survey respondents & the global programming community- Distribution of programming skills, experience and preferences- Employment-related information, preferences & opinionsLet's select a subset of columns with the relevant data for our analysis.<jupyter_code>selected_columns = [
    # Demographics
    'Country',
    'Age',
    'Gender',
    'EdLevel',
    'UndergradMajor',
    # Programming experience
    'Hobbyist',
    'Age1stCode',
    'YearsCode',
    'YearsCodePro',
    'LanguageWorkedWith',
    'LanguageDesireNextYear',
    'NEWLearn',
    'NEWStuck',
    # Employment
    'Employment',
    'DevType',
    'WorkWeekHrs',
    'JobSat',
    'JobFactors',
    'NEWOvertime',
    'NEWEdImpt'
]
len(selected_columns)<jupyter_output><empty_output><jupyter_text>Let's extract a copy of the data from these columns into a new data frame `survey_df`, which we can continue to modify further without affecting the original data frame.<jupyter_code>survey_df = survey_raw_df[selected_columns].copy()
schema = schema_raw[selected_columns]
survey_df
schema
schema.shape<jupyter_output><empty_output><jupyter_text>Let's view some basic information about the data frame.<jupyter_code>survey_df.shape
survey_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 64461 entries, 0 to 64460
Data columns (total 20 columns):
 #   Column                  Non-Null Count  Dtype  
---  ------                  --------------  -----  
 0   Country                 64072 non-null  object 
 1   Age                     45446 non-null  float64
 2   Gender                  50557 non-null  object 
 3   EdLevel                 57431 non-null  object 
 4   UndergradMajor          50995 non-null  object 
 5   Hobbyist                64416 non-null  object 
 6   Age1stCode              57900 non-null  object 
 7   YearsCode               57684 non-null  object 
 8   YearsCodePro            46349 non-null  object 
 9   LanguageWorkedWith      57378 non-null  object 
 10  LanguageDesireNextYear  54113 non-null  object 
 11  NEWLearn                56156 non-null  object 
 12  NEWStuck                54983 non-null  object 
 13  Employment              63854 non-null  object 
 14  DevType                 49370 non-null[...]<jupyter_text>Most columns have the data type `object`, either because they contain values of different types, or they contain empty values, which are represented using `NaN`. It appears that every column contains some empty values, since the Non-Null count for every column is lower than the total number of rows (64461). We'll need to deal with empty values and manually adjust the data type for each column on a case-by-case basis. Only two of the columns were detected as numeric columns (`Age` and `WorkWeekHrs`), even though there are a few other columns which have mostly numeric values. To make our analysis easier, let's convert some other columns into numeric data types, while ignoring any non-numeric value (they will get converted to NaNs)<jupyter_code>survey_df.Age1stCode
survey_df.Age1stCode.unique()
survey_df['Age1stCode'] = pd.to_numeric(survey_df.Age1stCode, errors='coerce')
survey_df['YearsCode'] = pd.to_numeric(survey_df.YearsCode, errors='coerce')
survey_df['YearsCodePro'] = pd.to_numeric(survey_df.YearsCodePro, errors='coerce')<jupyter_output><empty_output><jupyter_text>Let's now view some basic statistics about the the numeric columns.<jupyter_code>survey_df.describe()<jupyter_output><empty_output><jupyter_text>There seems to be a problem with the age column, as the minimum value is 1 and max value is 279. This is a common issues with surveys: responses may contain invalid values due to accidental or intentional errors while responding. A simple fix would be ignore the rows where the value in the age column is higher than 100 years or lower than 10 years as invalid survey responses. This can be done using the `.drop` method, [as explained here](https://www.geeksforgeeks.org/drop-rows-from-the-dataframe-based-on-certain-condition-applied-on-a-column/).<jupyter_code>survey_df.drop(survey_df[survey_df.Age < 10].index, inplace=True)
survey_df.drop(survey_df[survey_df.Age > 100].index, inplace=True)<jupyter_output><empty_output><jupyter_text>The same hold true for `WorkWeekHrs`. Let's ignore entries where the value for the column is higher than 140 hours. (~20 hours per day).<jupyter_code>survey_df.drop(survey_df[survey_df.WorkWeekHrs > 140].index, inplace=True)<jupyter_output><empty_output><jupyter_text>The gender column also allows picking multiple options, but to simplify our analysis, we'll remove values containing more than option.<jupyter_code>schema.Gender
survey_df['Gender'].value_counts()
import numpy as np
survey_df.where(~(survey_df.Gender.str.contains(';', na=False)), np.nan, inplace=True)<jupyter_output><empty_output><jupyter_text>We've now cleaned up and prepared the dataset for analysis. Let's take a look at sample of rows from the data frame.<jupyter_code>survey_df.sample(10)<jupyter_output><empty_output><jupyter_text>Let's save and commit our work before continuing.<jupyter_code>import jovian
jovian.commit()<jupyter_output><empty_output><jupyter_text>Exploratory Analysis and VisualizationBefore we can ask interesting questions about the survey responses, it would help to understand what the demographics i.e. country, age, gender, education level, employment level etc. of the respondents look like. It's important to explore these variable in order to understand how representative the survey is of the worldwide programming community, as a survey of this scale generally tends to have some [selection bias](https://en.wikipedia.org/wiki/Selection_bias).Let's begin by importing `matplotlib.pyplot` and `seaborn`.<jupyter_code>import seaborn as sns
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

sns.set_style('darkgrid')
matplotlib.rcParams['font.size'] = 14
matplotlib.rcParams['figure.figsize'] = (9, 5)
matplotlib.rcParams['figure.facecolor'] = '#00000000'<jupyter_output>/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm<jupyter_text>CountryLet's look at the number of countries from which there are responses in the survey, and plot the 10 countries with the highest number of responses.<jupyter_code>schema.Country
survey_df.Country.nunique()<jupyter_output><empty_output><jupyter_text>We can identify the countries with the highest number of respondents using the `value_counts` method.<jupyter_code>survey_df.Country.value_counts()
top_countries = survey_df.Country.value_counts().head(15)
top_countries<jupyter_output><empty_output><jupyter_text>We can visualize this information using a bar chart.<jupyter_code>plt.figure(figsize=(12,6))
plt.xticks(rotation=75)
plt.title(schema.Country)
sns.barplot(top_countries.index, top_countries);<jupyter_output><empty_output><jupyter_text>It appears the that a disproportionately high number of respondents are from USA & India - which one might expect since these countries have the highest populations (apart from China), and since the Survey is in English, which is the common language used by professionals in US, India & UK. We can already see that the survey may not be representative of the entire programming community - especially from non-English speaking countries.**Exercise**:Try finding the percentage of responses from English-speaking vs. non-English speaking countries. You can use [this list of languages spoken in different countries](https://github.com/JovianML/opendatasets/blob/master/data/countries-languages-spoken/countries-languages.csv).  AgeThe distribution of the age of respondents is another important factor to look at, and we can use a histogram to visualize it.<jupyter_code>schema.Age
plt.figure(figsize=(12, 6))
plt.title(schema.Age)
plt.xlabel('Age')
plt.ylabel('Number of respondents')

plt.hist(survey_df.Age, bins=np.arange(10,100,5), color='purple');<jupyter_output><empty_output><jupyter_text>It appears that a large percentage of respondents are in the age range of 20-45, which is somewhat representative of the programming community in general, as a lot of young people have taken up computer as their field of study or profession in the last 20 years.**Exercise**: You may want to filter out responses by age (or age group), if you'd like to analyze and compare the results of the survey for different age groups. Create a new column called `AgeGroup` which contains values like `Less than 10 years`, `10-18 years`, `18-30 years`, `30-45 years`, `45-60 years`, `Older than 60 years`, and repeat the analysis in the rest of this notebook for each age group.  GenderLet's look at the distribution of responses for the Gender. It's a well known fact that women and non-binary genders are underrepresented in the programming community, so we might expect to see a skewed distribution here.<jupyter_code>schema.Gender
gender_counts = survey_df.Gender.value_counts()
gender_counts
plt.figure(figsize=(12,6))
plt.title(schema.Gender)
plt.pie(gender_counts, labels=gender_counts.index, autopct='%1.1f%%', startangle=180);
gender_counts = survey_df.Gender.value_counts(dropna=False)
gender_counts<jupyter_output><empty_output><jupyter_text>A pie chart would be a good way to visualize the distribution.<jupyter_code>plt.figure(figsize=(12,6))
plt.title(schema.Gender)
plt.pie(gender_counts, labels=gender_counts.index, autopct='%1.1f%%', startangle=180);<jupyter_output><empty_output><jupyter_text>Only about 8% of survey respondents who have answered the question identify as women or non-binary. This number is lower than the overall percentage of women & non-binary genders in the programming community - which is estimated to be around 12%. **Exercise**: It would be interesting to compare the survey responses & preferences across genders, and you are encouraged to repeat this analysis with these breakdowns. How do the relative education levels differ across genders? How to salaries differ? You may find this analysis on the [Gender Divide in Data Science](https://medium.com/datadriveninvestor/exploratory-data-analysis-eda-understanding-the-gender-divide-in-data-science-roles-9faa5da44f5b) useful.  Education LevelFormal education in computer science is often considered an important requirement of becoming a programmer. Let's see if this indeed the case, especially since there are many free resources & tutorials available online to learn programming. We'll use a horizontal bar plot to compare education levels of respondents.<jupyter_code>schema.EdLevel
sns.countplot(y=survey_df.EdLevel)
plt.xticks(rotation=75);
plt.title(schema['EdLevel'])
plt.ylabel(None);<jupyter_output><empty_output><jupyter_text>It appears that well over half of the respondents hold a bachelor's or master's degree, so most programmers definitely seem to have some college education, although it's not clear from this graph alone if they hold a degree in computer science.**Exercises**: The graph currently shows the number of respondents for each option, can you modify it to show the percentage instead? Further, can you break down the graph to compare the percentages for each degree for men vs. women.  Let's also plot undergraduate majors, but this time we'll convert the numbers into percentages, and sort by percentage values to make it easier to visualize the order.<jupyter_code>schema.UndergradMajor
undergrad_pct = survey_df.UndergradMajor.value_counts() * 100 / survey_df.UndergradMajor.count()

sns.barplot(undergrad_pct, undergrad_pct.index)

plt.title(schema.UndergradMajor)
plt.ylabel(None);
plt.xlabel('Percentage');<jupyter_output><empty_output><jupyter_text>It turns that 40% of programmers holding a college degree have a field of study other than computer science - which is very encouraging. This seems to suggest that while college education is helpful in general, you do not need to pursue a major in computer science to become a successful programmer.**Exercises**: Analyze the results of the `NEWEdImpt` column for respondents who hold some college degree vs. those who don't. Do you notice any difference in opinion?<jupyter_code>schema.NEWEdImpt<jupyter_output><empty_output><jupyter_text>EmploymentFreelancing or contract work is a common choice among programmer, so it would be interesting to compare the breakdown between full time, part time & freelance work. Let's visualize the data from `Employment` column.<jupyter_code>schema.Employment
(survey_df.Employment.value_counts(normalize=True, ascending=True)*100).plot(kind='barh', color='g')
plt.title(schema.Employment)
plt.xlabel('Percentage');<jupyter_output><empty_output><jupyter_text>It appears that close to 10% of respondents are employed part time or as freelancers.**Exercise**: Add a new column `EmploymentType` which contains values `Enthusiast` (student or not employed but looking for work), `Professional` (employed full-time, part-time or freelancing) and `Other` (not employed or retired). For each of the graphs that follow, show a comparison between `Enthusiast` and `Professional`. The `DevType` field contains information about the roles held by respondents. Since the question allows multiple answers, the column contains lists of values separated by `;`, which makes it a bit harder to analyze directly.<jupyter_code>schema.DevType
survey_df.DevType.value_counts()<jupyter_output><empty_output><jupyter_text>Let's define a helper function which turns a column containing lists of values (like `survey_df.DevType`) into a data frame with one column for each possible option.<jupyter_code>def split_multicolumn(col_series):
    result_df = col_series.to_frame()
    options = []
    # Iterate over the column
    for idx, value  in col_series[col_series.notnull()].iteritems():
        # Break each value into list of options
        for option in value.split(';'):
            # Add the option as a column to result
            if not option in result_df.columns:
                options.append(option)
                result_df[option] = False
            # Mark the value in the option column as True
            result_df.at[idx, option] = True
    return result_df[options]
dev_type_df = split_multicolumn(survey_df.DevType)
dev_type_df<jupyter_output><empty_output><jupyter_text>The `dev_type_df` has one column for each option that can be selected as a response. If a responded has selected the option, the value in the column is `True`, otherwise it is false.We can now use the column-wise totals to identify the most common roles.<jupyter_code>dev_type_totals = dev_type_df.sum().sort_values(ascending=False)
dev_type_totals<jupyter_output><empty_output><jupyter_text>As one might expect, the most common roles include "Developer" in the name. **Exercises**: * Can you figure out what percentage of respondents work in roles related to data science? * Which role has the highest percentage of women? We've only explore a handful of columns from the 20 columns that we selected. Explore and visualize the remaining columns using the empty cells below.<jupyter_code><jupyter_output><empty_output><jupyter_text>Let's save and upload our work before continuing.<jupyter_code>import jovian
jovian.commit()<jupyter_output><empty_output><jupyter_text>Asking and Answering QuestionsWe've already gained several insights about the respondents and the programming community in general, simply by exploring individual columns of the dataset. Let's ask some specific questions, and try to answer them using data frame operations and interesting visualizations.  Q: Which were the most popular programming languages in 2020? To answer, this we can use the `LanguageWorkedWith` column. Similar to `DevType` respondents were allowed to choose multiple options here.<jupyter_code>schema.LanguageWorkedWith
schema.LanguageDesireNextYear
survey_df.LanguageWorkedWith<jupyter_output><empty_output><jupyter_text>First, we'll split this column into a data frame containing a column of each languages listed in the options.<jupyter_code>languages_worked_df = split_multicolumn(survey_df.LanguageWorkedWith)
languages_worked_df<jupyter_output><empty_output><jupyter_text>It appears that a total of 25 languages were included among the options. Let's aggregate these to identify the percentage of respondents who selected each language.<jupyter_code>languages_worked_percentages = languages_worked_df.mean().sort_values(ascending=False) * 100
languages_worked_percentages<jupyter_output><empty_output><jupyter_text>We can plot this information using a horizontal bar chart.<jupyter_code>plt.figure(figsize=(12, 12))
sns.barplot(languages_worked_percentages, languages_worked_percentages.index)
plt.title("Languages used in the past year");
plt.xlabel('count');<jupyter_output><empty_output><jupyter_text>Perhaps not surprisingly, Javascript & HTML/CSS comes out at the top as web development is one of the most sought skills today and it's also happens to be one of the easiest to get started with. SQL is necessary for working with relational databases, so it's no surprise that most programmers work with SQL on a regular basis. For other forms of development, Python seems be the popular choice, beating out Java, which was the industry standard for server & application development for over 2 decades.**Exercises**:* What are the most common languages used by students? How does the list compare with the most common languages used by professional developers?* What are the most common languages among respondents who do not describe themselves as "Developer, front-end"?* What are the most common languages among respondents who work in fields related to data science?* What are the most common languages used by developers older than 35 years of age? * What are the most common languages used by developers in your home country?  Q: Which languages are the most people interested to learn over the next year?For this we can can use the `LanguageDesireNextYear` column, with similar processing as the previous one.<jupyter_code>schema.LanguageDesireNextYear
languages_interested_df = split_multicolumn(survey_df.LanguageDesireNextYear)
languages_interested_percentages = languages_interested_df.mean().sort_values(ascending=False) * 100
languages_interested_percentages
plt.figure(figsize=(12, 12))
sns.barplot(languages_interested_percentages, languages_interested_percentages.index)
plt.title("Languages people are intersted in learning over the next year");
plt.xlabel('count');<jupyter_output><empty_output><jupyter_text>Once again, it's not surprising that Python is the language most people are interested in learning - since it is an easy-to-learn general purpose programming language well suited for a variety of domains: application development, numerical computing, data analysis, machine learning, big data, cloud automation, web scraping, scripting etc. etc. We're using Python for this very analysis, so we're in good company!**Exercises**: Repeat all the exercises for the previous question, replacing "most common languages" with "languages people are interested in learning/using".  Q:  Which are the most loved languages i.e. a high percentage of people who have used the language want to continue learning & using it over the next year?While this question may seem trick at first, it's really easy to solve using Pandas array operations. Here's what we can do:- Create a new data frame `languages_loved_df` which contains a `True` value for a language only if the corresponding values in `languages_worked_df` and `languages_interested_df` are both `True`- Take the column wise sum of `languages_loved_df` and divide it by the column-wise sum of `languages_worked_df` to get the percentage of respondents who "love" the language- Sort the results in decreasing order and plot a horizontal bar graph<jupyter_code>True & False
languages_loved_df = languages_worked_df & languages_interested_df
languages_loved_df
languages_loved_percentages = (languages_loved_df.sum() * 100/ languages_worked_df.sum()).sort_values(ascending=False)
languages_loved_percentages
plt.figure(figsize=(12, 12))
sns.barplot(languages_loved_percentages, languages_loved_percentages.index)
plt.title("Most loved languages");
plt.xlabel('count');<jupyter_output><empty_output><jupyter_text>[Rust](https://www.rust-lang.org) has been StackOverflow's most-loved language for [4 years in a row](https://stackoverflow.blog/2020/01/20/what-is-rust-and-why-is-it-so-popular/), followed by TypeScript which has gained a lot of popularity in the past few years as a good alternative to JavaScript for web development.Python features at number 3, despite already being one of the most widely-used languages in world. This is testament to the fact the language has solid foundation, is really easy to learn & use, has a strong ecosystem of libraries for various and massive worldwide community of developers to enjoy using it.**Exercises:** What are the _most dreaded languages_ i.e. languages which people have used in the past year, but do not want to learn/use over the next year. Hint: `~languages_interested_df`  Q: In which countries do developers work the highest number of hours per week? Consider countries with more than 250 responses only.To answer this question, we'll need to use the `groupby` data frame method to aggregate the rows for each country. We'll also need to filter the results to only include the countries which have more than 250 respondents.<jupyter_code>survey_df.groupby('Country')[['WorkWeekHrs', 'Age']].mean()
countries_df = survey_df.groupby('Country')[['WorkWeekHrs']].mean().sort_values('WorkWeekHrs', ascending=False)
high_response_countries_df = countries_df.loc[survey_df.Country.value_counts() > 250].head(15)
high_response_countries_df<jupyter_output><empty_output><jupyter_text>The Asian countries like Iran, China & Israel have the highest working hours, followed by the United States. However, there isn't too much variation overall and the average working hours seem to be around 40 hours per week.**Exercises:*** How to the average work hours compare across continents? You may find this list of [countries in each continent](https://hub.jovian.ml/wp-content/uploads/2020/09/countries.csv) useful.* Which role has the highest average number of hours worked per week? Which role has the lowest?* How do the hours worked compare between freelancers and developers working full-time?  Q: How important is it to start young to build a career in programming?Let's create a scatter plot of `Age` vs. `YearsCodePro` (i.e. years of coding experience) to answer this question.<jupyter_code>schema.YearsCodePro
sns.scatterplot('Age', 'YearsCodePro', hue='Hobbyist', data=survey_df)
plt.xlabel("Age")
plt.ylabel("Years of professional coding experience");<jupyter_output><empty_output><jupyter_text>You can see points all over the graph, which seems to indicate that you can **start programming professionally at any age**. Also, many people who have been coding for several decades professionally also seem to enjoy it has a hobby.We can also view the distribution of `Age1stCode` column to see when the respondents tried programming for the first time.<jupyter_code>plt.title(schema.Age1stCode)
sns.distplot(survey_df.Age1stCode);<jupyter_output><empty_output><jupyter_text>As you might expect, most people seem to have had some exposure to programming before the age of 40, but there are people of all ages and walks of life who are learning to code.**Exercises**:* How does experience change opinions & preferences? Repeat the entire analysis while comparing the responses of people who have more than 10 years of professional programming experience v.s. those who don't. Do you see any interesting trends?* Compare the years of professional coding experience across different genders.  We've barely scratched the surface here, and hopefully you are already thinking of many more questions that you'd like to answer using this data. Use the empty cells to below to ask and answer more questions.<jupyter_code><jupyter_output><empty_output><jupyter_text>Let's save and commit our work before continuing<jupyter_code>import jovian
jovian.commit()<jupyter_output><empty_output><jupyter_text>Inferences and ConclusionsWe've drawn many interesting inferences from the survey, here's a summary of the few of them:- Based on the demographics of the survey respondents, we can infer that the survey is somewhat representative of the overall programming community, although it definitely has fewer responses from programmers in non-English-speaking countries and from women & non-binary genders.- The programming community is not as diverse as it can be, and although things are improving, we should take more efforts to support & encourage members of underrepresented communities - whether it is in terms of age, country, race, gender or otherwise.- Most programmers hold a college degree, although a fairly large percentage did not have computer science as their major in college, so a computer science degree isn't compulsory for learning to code or building a career in programming.- A significant percentage of programmers either work part time or as freelancers, and this can be a great way to break into the field, especially when you're just getting started.- Javascript & HTML/CSS are the most used programming languages in 2020, closely followed by SQL & Python- Python is the language most people are interested in learning - since it is an easy-to-learn general purpose programming language well suited for a variety of domains.- Rust and TypeScript are the most "loved" languages in 2020, both of which have small but fast-growing communities. Python is a close third, despite already being a widely used language.- Programmers around the world seems be working for around 40 hours a week on average, with slight variations by country.- You can learn and start programming professionally at any age, and you're likely to have a long and fulfilling career if you also enjoy programming as a hobby.  References and Future WorkThere's a wealth of information to be discovered using the survey, and we've barely scratched the surface. Here are some ideas for further exploration:- Repeat the analysis for different age groups & genders, and compare the results- Choose a different set of columns (we chose on 20 out of 65) to analyze other facets of the data- Prepare an analysis focusing on diversity - and identify areas where underrepresented communities are at par with the majority (e.g. education) and where they aren't (e.g. salaries)- Compare the results of this year's survey with the previous years and identify interesting trendsReferences:- Stack Overflow Developer Survey: https://insights.stackoverflow.com/survey- Pandas user guide: https://pandas.pydata.org/docs/user_guide/index.html- Matplotlib user guide: https://matplotlib.org/3.3.1/users/index.html- Seaborn user guide & tutorial: https://seaborn.pydata.org/tutorial.html- `opendatasets` Python library: https://github.com/JovianML/opendatasetsAs a next step, you can try out a project on another dataset of your choice: https://jovian.ml/aakashns/zerotopandas-course-project-starter<jupyter_code>import jovian
jovian.commit()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07122004957312035, 'fraction_numerical': 0.010548058386119526, 'mean_word_length': 5.3984140969163, 'pattern_counts': {'":': 0, '<': 181, '<?xml version=': 0, '>': 186, 'https://': 33, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '45', 'dir': 'jupyter-structured-clean-dedup', 'id': '591336', 'license': 'MIT', 'n_tokens_mistral': 10257, 'n_tokens_neox': 9578, 'n_words': 4187, 'path': 'Lecture 6/Notebook-9-python-eda-stackoverflow-survey.ipynb', 'repo_name': 'FabianAC07/Data-Analysis-with-Python-Zero-to-Pandas', 'size': '36310'}