<jupyter_start><jupyter_text>**Filter**- Keep only English language records.- Remove records that contain exactly 50 citations (I had plotted a histograms and it seems there was some discrepancy with this value -- better just remove it)**Transform**- Add an extra column for age.- Add an extra column for not containing any semantic words.<jupyter_code>print (len(df_dblp[(df_dblp["lang"] == "en") & (df_dblp["indeg"] != 50)]))
df_dblp = df_dblp[(df_dblp["lang"] == "en") & (df_dblp["indeg"] != 50)]
df_dblp["age"] = 2019 - df_dblp["year"]
df_dblp["isUNK"] = (df_dblp["max_prog_word"] == "UNK").astype(int)
# for simplicity of descriptive analysis, let's remove docs which don't hit the lexicon
# In the regression we will have to add a dummy variable to account for this.
drop_misses = True
if drop_misses:
    df_dblp = df_dblp[df_dblp["isUNK"] == 0]
print (len(df_dblp))<jupyter_output>1511088<jupyter_text>**Descriptive analysis**<jupyter_code># quartile-ish bins of the innovation feature  
df_dblp['max_prog_q'] = pd.qcut(df_dblp['max_prog'],4, labels=["<25", "<50", "<75", "<100"], duplicates='drop')
# custom bins of the innovation feature  
bins = pd.IntervalIndex.from_tuples([(0, 1), (1, 3), (3, 20)], closed="left")
df_dblp['nprog95_q'] = pd.cut (df_dblp["nprog95"], bins=bins, labels=["0", "<=2", "3+"], retbins=False, duplicates="drop")
sns.set_context("paper")
fig,ax = plt.subplots(1,1,figsize=[7,2],sharey=True)

#sns.barplot(y='indeg',x='sem_q',data=df_ops,ax=ax[0]);
#ax[0].set_title('Legal opinions')
#ax[0].set_xlabel('Quantiles')
#ax[0].set_ylabel('Citations')
sns.barplot(y='indeg',x='max_prog_q',data=df_dblp,ax=ax);
ax.set_title('Scientific articles');
ax.set_xlabel('Quantiles')
ax.set_ylabel('Citations')<jupyter_output><empty_output><jupyter_text>This does not look great. But we have seen this before. There are two possible explanations for this:- DBLP contains a lot of seminal old papers which may have gotten popular due to other reasons besides progressiveness. Moreover, they have more coverage of papers in the recent years but contain only the most popular ones from the past.- The more recent papers are highly progressive but haven't had much time to have an impact.Let's just control for age and see what the story is.<jupyter_code>df_dblp_post_2010 = df_dblp[(df_dblp["year"] > 2010)]
df_dblp_post_2000 = df_dblp[(df_dblp["year"] > 2000)]
df_dblp_post_1990 = df_dblp[(df_dblp["year"] > 1990)]

sns.set_context("paper")
fig,ax = plt.subplots(1,3,figsize=[7,2],sharey=True)

sns.barplot(y='indeg',x='max_prog_q',data=df_dblp_post_2010,ax=ax[0]);
ax[0].set_title('Post 2010')
ax[0].set_xlabel('Quantiles')
ax[0].set_ylabel('Citations')

sns.barplot(y='indeg',x='max_prog_q',data=df_dblp_post_2000,ax=ax[1]);
ax[1].set_title('Post 2000');
ax[1].set_xlabel('Quantiles')
ax[1].set_ylabel('')

sns.barplot(y='indeg',x='max_prog_q',data=df_dblp_post_1990,ax=ax[2]);
ax[2].set_title('Post 1990');
ax[2].set_xlabel('Quantiles')
ax[2].set_ylabel('')<jupyter_output><empty_output><jupyter_text>Looks much better.- Post 2010, we see the expected trend ruling out the second explanation for the discrepancy.- Post 2000, almost even.- Post 1990, the trend reverses which aligns with the intuition that older papers get more popular because of their age.Its still not ideal, but hopefully by controlling for age of the document in the multivariate analysis, we will get the right answer.(perhaps also points to the limitation of our metric which tends to score more heavily on progressiveness for recent documents)<jupyter_code>df_dblp_2000s = df_dblp[(df_dblp["year"] > 1980) & (df_dblp["year"] <= 2010)]
print (len(df_dblp_2000s))
sns.set_context("paper")
fig,ax = plt.subplots(1,1,figsize=[6,3.5])
sns.barplot(y='indeg',x='max_prog_q',data=df_dblp_2000s,ax=ax);
ax.set_title('Scientific articles');
pd.IntervalIndex.from_tuples?
bins = pd.IntervalIndex.from_tuples([(0, 1), (1, 4), (4, np.inf)], closed="left")
df_dblp['nprog50_q'] = pd.cut (df_dblp["nprog50"], bins=bins, labels=["0", "<=2", "3+"], retbins=False, duplicates="drop")
df_dblp['nprog90_q'] = pd.cut (df_dblp["nprog90"], bins=bins, labels=["0", "<=2", "3+"], retbins=False, duplicates="drop")
df_dblp['nprog95_q'] = pd.cut (df_dblp["nprog95"], bins=bins, labels=["0", "<=2", "3+"], retbins=False, duplicates="drop")
df_dblp['nprog99_q'] = pd.cut (df_dblp["nprog99"], bins=bins, labels=["0", "<=2", "3+"], retbins=False, duplicates="drop")
df_dblp_2000s = df_dblp[(df_dblp["year"] > 1980) & (df_dblp["year"] <= 2010)]
print (len(df_dblp_2000s))
sns.set_context("paper")
fig,ax = plt.subplots(1,1,figsize=[6,3.5])
sns.barplot(y='indeg',x='nprog95_q',data=df_dblp_2000s,ax=ax);
ax.set_title('Scientific articles');
df_dblp_2000s.columns.values
sns.set_context("paper")
fig,ax = plt.subplots(1,4,figsize=[10,2],sharey=True)

sns.barplot(y='indeg',x='nprog50_q',data=df_dblp_2000s,ax=ax[0]);
ax[0].set_title('>50 percentile')
ax[0].set_xlabel('Intervals')
ax[0].set_ylabel('Citations')


sns.barplot(y='indeg',x='nprog90_q',data=df_dblp_2000s,ax=ax[1]);
ax[1].set_title('>90 percentile');
ax[1].set_xlabel('Intervals')
ax[1].set_ylabel('')

sns.barplot(y='indeg',x='nprog95_q',data=df_dblp_2000s,ax=ax[2]);
ax[2].set_title('>95 percentile');
ax[2].set_xlabel('Intervals')
ax[2].set_ylabel('')

sns.barplot(y='indeg',x='nprog99_q',data=df_dblp_2000s,ax=ax[3]);
ax[3].set_title('>99 percentile');
ax[3].set_xlabel('Intervals')
ax[3].set_ylabel('')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.18235078725741485, 'fraction_numerical': 0.049066276089344565, 'mean_word_length': 8.196969696969697, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '482174', 'license': 'MIT', 'n_tokens_mistral': 2395, 'n_tokens_neox': 2154, 'n_words': 435, 'path': 'notebooks/DBLP.univariate.ipynb', 'repo_name': 'sandeepsoni/semantic-progressiveness', 'size': '5462'}