<jupyter_start><jupyter_text>Is there a correlation between programming experience and programming hobby? Let's explore.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import YearsAndHobby as yh
%matplotlib inline

df = pd.read_csv('./survey_results_public.csv')
print("The total number of rows in data set is {}.".format(df.shape[0]))

# Check the column YearsProgram for missing values.
print("The number of rows with invalid YearsProgram is {}."
      .format(sum(df['YearsProgram'].isnull())))
# Fill in the missing values.
yp_median = yh.get_median_of_years(
    df[df['YearsProgram'].isnull()==False]['YearsProgram'])
print(yp_median)
years_program_df = df.copy()
years_program_df['YearsProgram'].fillna(yp_median, inplace=True)
print(sum(years_program_df['YearsProgram'].isnull()))
# Let's take a look at the counts and ratios of people 
# with different programming experiences.
counts_df = yh.gen_counts_df_by_years(
    'YearsProgram', years_program_df)
counts = [
    sum(counts_df['count']),
    sum(counts_df['dev_count']),
    sum(counts_df['not_dev_count'])
]
sums = [
    sum(counts_df['ratio']),
    sum(counts_df['dev_ratio']),
    sum(counts_df['not_dev_ratio'])
]
[counts, sums]
counts_df
# Let's draw the graph.
x = np.arange(len(counts_df['years']))
width = 0.25

# Draw a bar chart.
plt.bar(x=x, height=counts_df['not_dev_ratio'], width=width, color='red', label=u'not dev')
plt.bar(x=x+width, height=counts_df['dev_ratio'], width=width, color='lightblue', label=u'dev')
plt.bar(x=x+width*2, height=counts_df['ratio'], width=width, color='green', label=u'overall')
plt.xticks(x, counts_df['years'], size='small',rotation=90)
plt.xlabel('YearsProgram')
plt.ylabel('Not Dev/Dev/Overall ratio')
plt.title('Programming Experience of Users')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Overall, most of the sample users in StackOverflow's programming experience is within 1-8 years. However, many of the programming experience of non-professional developers is within 5 years, and the proportion is very large. For professional developers, most of their programming experience is 2 years to 10, and there are many more people with more experience. I want to focus on professional developers and want to see the impact of programming hobby on their careers. So, I need to sort out the relevant data.<jupyter_code>dev_hobby_ratios_df = yh.gen_hobby_ratios_df_by_years(
    'YearsProgram', years_program_df, only_dev=True)
dev_hobby_ratios_df
# Let's draw the graph.
x = np.arange(len(counts_df['years']))

# Draw a line chart.
plt.plot(x, dev_hobby_ratios_df['hobby_ratio'], 'yellow')
plt.plot(x, dev_hobby_ratios_df['contrib_ratio'], 'red')
plt.xticks(x, counts_df['years'], size='small',rotation=90)
plt.xlabel('YearsProgram')
plt.ylabel('Hobby/Contrib ratio')
plt.title('Hobby ratio with Programming Experience')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>As we can see, the proportion of professional developers who love programming does not increase with the increase in programming experience. Overall, while the curve on the contribution ratio of open source projects has risen slightly, it doesn't mean anything. It may not be so appropriate to choose feature YearsProgram here. Let's try again with feature YearsCodedJob.<jupyter_code>print("The number of rows with invalid YearsCodedJob is {}."
      .format(sum(df['YearsCodedJob'].isnull())))
# Discard rows with missing values.
# Because if we fill in the missing values with the median, 
# it will have a serious impact on the following analysis.
years_coded_job_df = df[df['YearsCodedJob'].isnull()==False]
counts_df2 = yh.gen_counts_df_by_years('YearsCodedJob', years_coded_job_df)

x = np.arange(len(counts_df['years']))
width = 0.25

# Draw a bar chart.
plt.bar(x=x, height=counts_df2['not_dev_ratio'], width=width, color='red', label=u'not dev')
plt.bar(x=x+width, height=counts_df2['dev_ratio'], width=width, color='lightblue', label=u'dev')
plt.bar(x=x+width*2, height=counts_df2['ratio'], width=width, color='green', label=u'overall')
plt.xticks(x, counts_df2['years'], size='small',rotation=90)
plt.xlabel('YearsCodedJob')
plt.ylabel('Not Dev/Dev/Overall ratio')
plt.title('Coded Job Experience of Users')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Obviously, from this point of view, most sample users have less experience in coded job.<jupyter_code>dev_hobby_ratios_df2 = yh.gen_hobby_ratios_df_by_years(
    'YearsCodedJob', years_coded_job_df, only_dev=True)

x = np.arange(len(counts_df['years']))

# 绘制折线图。
# Draw a line chart.
plt.plot(x, dev_hobby_ratios_df2['hobby_ratio'], 'yellow')
plt.plot(x, dev_hobby_ratios_df2['contrib_ratio'], 'red')
plt.xticks(x, counts_df['years'], size='small',rotation=90)
plt.xlabel('YearsCodedJob')
plt.ylabel('Hobby/Contrib ratio')
plt.title('Hobby ratio with Coded Job Experience')
plt.legend()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13128266988339365, 'fraction_numerical': 0.006232408524326498, 'mean_word_length': 6.921974522292993, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '337114', 'license': 'FTL', 'n_tokens_mistral': 1777, 'n_tokens_neox': 1682, 'n_words': 436, 'path': '2_YearsAndHobby.ipynb', 'repo_name': 'hyper0x/DSND_T2_P1_Public', 'size': '4974'}