<jupyter_start><jupyter_text>StackOverflow Insights By <NAME> and <NAME> Data Source:StackLite by StackOverflow (Version 4)https://www.kaggle.com/stackoverflow/stacklitehttps://data.stackexchange.com/stackoverflow/query/637401/question-text-for-ruby-on-rails<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
question_tags = pd.read_csv('question_tags.csv')
len(question_tags)
question_tags.head(1)
questions = pd.read_csv('questions.csv')
len(questions)
questions.head(1)<jupyter_output><empty_output><jupyter_text>Process Data<jupyter_code># Convert dates to datetime objects
dt_creation = pd.to_datetime(questions['CreationDate'])
dt_closed = pd.to_datetime(questions['ClosedDate'])
dt_deletion = pd.to_datetime(questions['DeletionDate'])
q_creation = questions.assign(creation = dt_creation)
q_closed = q_creation.assign(closed = dt_closed)
q_with_date = q_closed.assign(deletion = dt_deletion)
q_with_date.head(1)<jupyter_output><empty_output><jupyter_text>Throughout the years...  1) Number of questions<jupyter_code>q_year = q_with_date
q_year['creationyear'] = q_with_date['creation'].apply(lambda x: x.year)
q_each_year = q_year['creationyear'].value_counts().sort_index()
q_by_year = q_each_year.plot(kind='bar', title='Total number of questions each year')
q_by_year.set_xlabel("Year")
q_by_year.set_ylabel("Number of questions")
q_by_year<jupyter_output><empty_output><jupyter_text>**Observation:** Number of questions increased every year. Slowing growth in the past 2 years.**Remark:** 2008 is the year StackOverflow is founded. 2008 and 2017 not full years.<jupyter_code>q_year_closed = q_year[q_year['closed'].notnull()]
q_year_closed.head(1)
q_closed_each_year = q_year_closed['creationyear'].value_counts().sort_index()
q_by_year_closed = q_closed_each_year.plot(kind='bar', title='Number of closed questions each year')
q_by_year_closed.set_xlabel("Year")
q_by_year_closed.set_ylabel("Number of questions")
q_by_year_closed<jupyter_output><empty_output><jupyter_text>**Observation:** Closed questions (includes duplicates, opinion-based, etc) increased until 2013 when it is relatively stable after<jupyter_code>q_year_notclosed = q_year[q_year['closed'].isnull()]
q_year_open = q_year_notclosed[q_year_notclosed['deletion'].isnull()]
q_year_open.head(1)
q_open_each_year = q_year_open['creationyear'].value_counts().sort_index()
q_by_year_open = q_open_each_year.plot(kind='bar', title='Number of open questions each year')
q_by_year_open.set_xlabel("Year")
q_by_year_open.set_ylabel("Number of questions")
q_by_year_open<jupyter_output><empty_output><jupyter_text>2) Number of answers<jupyter_code>q_year_ans = q_year[['creationyear', 'AnswerCount']]
q_year_ans_mean = q_year_ans.groupby('creationyear').mean()
avg_anscount_year = q_year_ans_mean.plot(kind='bar', title='Average answer counts throughout the years', y='AnswerCount')
avg_anscount_year.set_xlabel("Year")
avg_anscount_year.set_ylabel("Average Number of Answers")
avg_anscount_year<jupyter_output><empty_output><jupyter_text>**Observations** Average number of questions have been declining each year  3) Scores<jupyter_code>q_year_score = q_year[['creationyear', 'Score']]
q_year_score_mean = q_year_score.groupby('creationyear').mean()
avg_score_year = q_year_score_mean.plot(kind='bar', title='Average scores throughout the years', y='Score')
avg_score_year.set_xlabel("Year")
avg_score_year.set_ylabel("Average score")
avg_score_year
q_year_score_mean_exc = q_year_score_mean.drop([2008,2017])
avg_score_year = q_year_score_mean_exc.plot(kind='bar', title='Average scores throughout the years (excluding 2008 and 2017)', y='Score')
avg_score_year.set_xlabel("Year")
avg_score_year.set_ylabel("Average score")
avg_score_year<jupyter_output><empty_output><jupyter_text>**Observation:** Average scores have been declining. This could be due to lower quality questions and answers or increased amount of questions or other reasons.  4) Tags<jupyter_code>cb = pd.concat([question_tags, q_with_date], axis=1)
tag_year = cb[['Tag','creationyear']]

# storing top 50 tags
top_tag = tag_year['Tag'].value_counts()
top_tag.head(50).to_csv("top_tag.csv")
# to see how each tag change over years
tag_by_year = tag_year.groupby(['Tag', 'creationyear']).size().reset_index(name="freq")
tag_by_year.sort_values('freq', ascending=0).head()
tag_by_year.sort_values(['freq'], ascending=0)

# take pandas for example, a better version of graph is done in Tableau
tag_by_year_pandas = tag_by_year.loc[tag_by_year['Tag'] == "pandas"]
#tag_by_year_pandas.plot(x='creationyear', y='freq')

tag_by_year_pandas.to_csv('pandas.csv')

tag_by_year_r = tag_by_year.loc[tag_by_year['Tag'] == "r"]
#tag_by_year_r.plot(x='creationyear', y='freq')

tag_by_year_r.to_csv('r.csv')

tag_by_year_ror = tag_by_year.loc[tag_by_year['Tag'] == "ruby-on-rails"]
#tag_by_year_ror.plot(x='creationyear', y='freq')

tag_by_year_ror.to_csv('ror.csv')<jupyter_output><empty_output><jupyter_text>Throughout the week...Day of the week [ 0 = Monday ; 6 = Sunday ]  1) Number of questions in 2016<jupyter_code>q_day_2016 = q_year[q_year['creationyear'] == 2016]
q_creation_day_2016 = q_day_2016['creation'].apply(lambda x: x.weekday())
q_day_2016 = q_day_2016.assign(creationday = q_creation_day_2016)
q_each_day_2016 = q_day_2016['creationday'].value_counts().sort_index()
q_by_day_2016 = q_each_day_2016.plot(kind='bar', title='Total number of questions each day in 2016')
q_by_day_2016.set_xlabel("Day of the week")
q_by_day_2016.set_ylabel("Number of questions")
q_by_day_2016<jupyter_output><empty_output><jupyter_text>throughout the years<jupyter_code># q_year_day = q_day.groupby(["creationyear", "creationday"]).size()
q_day = q_year
q_creation_day = q_day['creation'].apply(lambda x: x.weekday())
q_day = q_day.assign(creationday = q_creation_day)
q_year_day_pivot = q_day.pivot_table(index='creationday', columns='creationyear', values='Id', aggfunc=len)
q_year_day_pivot
q_year_day_pivot_exc = q_year_day_pivot.drop([2008, 2017], axis=1)
q_year_day_pivot_exc_plot = q_year_day_pivot_exc.plot(title='Number of questions each day')
q_year_day_pivot_exc_plot.set_xlabel("Day of Week")
q_year_day_pivot_exc_plot.set_ylabel("Number of questions")
q_year_day_pivot_exc_plot<jupyter_output><empty_output><jupyter_text>2) Number of answers in 2016<jupyter_code>q_day_ans_2016 = q_day_2016[['creationday', 'AnswerCount']]
q_day_ans_2016_mean = q_day_ans_2016.groupby('creationday').mean()
avg_anscount_day_2016 = q_day_ans_2016_mean.plot(kind='bar', title='Average answer counts daily in 2016', y='AnswerCount')
avg_anscount_day_2016.set_xlabel("Day of Week")
avg_anscount_day_2016.set_ylabel("Average Number of Answers")
avg_anscount_day_2016<jupyter_output><empty_output><jupyter_text>throughout the years<jupyter_code>q_day_ans = q_day[['creationyear', 'creationday', 'AnswerCount']]
q_day_ans_pivot = q_day.pivot_table(index='creationday', columns='creationyear', values='AnswerCount')
q_day_ans_pivot_exc = q_day_ans_pivot.drop([2008, 2017], axis=1)
q_day_ans_pivot_exc_plot = q_day_ans_pivot_exc.plot(title='Average number of answers each day')
q_day_ans_pivot_exc_plot.set_xlabel("Day of Week")
q_day_ans_pivot_exc_plot.set_ylabel("Average number of answers")
q_day_ans_pivot_exc_plot<jupyter_output><empty_output><jupyter_text>3) Scores in 2016<jupyter_code>q_day_score_2016 = q_day_2016[['creationday', 'Score']]
q_day_score_2016_mean = q_day_score_2016.groupby('creationday').mean()
avg_score_day_2016 = q_day_score_2016_mean.plot(kind='bar', title='Average score daily in 2016', y='Score')
avg_score_day_2016.set_xlabel("Day of Week")
avg_score_day_2016.set_ylabel("Average Score")
avg_score_day_2016<jupyter_output><empty_output><jupyter_text>throughout the years<jupyter_code>q_day_score = q_day[['creationyear', 'creationday', 'Score']]
q_day_score_pivot = q_day_score.pivot_table(index='creationday', columns='creationyear', values='Score')
q_day_score_pivot_exc = q_day_score_pivot.drop([2008, 2017], axis=1)
q_day_score_pivot_exc_plot = q_day_score_pivot_exc.plot(title='Average score each day')
q_day_score_pivot_exc_plot.set_xlabel("Day of Week")
q_day_score_pivot_exc_plot.set_ylabel("Average score")
q_day_score_pivot_exc_plot<jupyter_output><empty_output><jupyter_text>Throughout the day...<jupyter_code>ans_tod = q_with_date[['Id', 'AnswerCount', 'creation']]
# ans_tod['hour'] = ans_tod['creation'].apply(lambda x: x.hour)
ans_tod.groupby('hour').mean().plot()

ans_tod.groupby('hour').mean().to_csv("ans_by_tod.csv")<jupyter_output><empty_output><jupyter_text>How many tags should I use to get the most answers?<jupyter_code>cb.head()
tag_ans = cb[['Id', 'Tag', 'AnswerCount']]
tag_ans.head()
tag_ans.columns=['Id', 'dup', 'Tag', 'AnswerCount']
tag_ans = tag_ans[['Id', 'Tag', 'AnswerCount']]
q_tag_ans = tag_ans.groupby('Id').size().reset_index(name='TagCount')
q_tag_ans['AnswerCount'] = questions['AnswerCount']
q_tag_ans.groupby('TagCount')['AnswerCount'].mean().plot()
q_tag_ans.groupby('TagCount')['AnswerCount'].mean().to_csv("answerByTags.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13244161824361364, 'fraction_numerical': 0.026312904286810657, 'mean_word_length': 10.431077694235588, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 67, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '470128', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 3661, 'n_tokens_neox': 3367, 'n_words': 532, 'path': 'stacklite.ipynb', 'repo_name': 'ningjung/stacklite364', 'size': '9142'}