<jupyter_start><jupyter_text>Exploratory Data Analysis  Author: <NAME> Date: 6/21/2021<jupyter_code>#start working with the smaller dataset
#departments, agencies
#look into the project titles? Ask for guidance if needed
#focus on agency, department, FY, project start year, budget start year, total cost, subproject cost.  Also, which projects have a keyword list? Broken down by agency?<jupyter_output><empty_output><jupyter_text>plotting histogram plt.hist(df['base_price'],rwidth=0.9,alpha=0.3,color='blue',bins=15,edgecolor='red') x and y-axis labels plt.xlabel('Base price range') plt.ylabel('Distinct order') plot title plt.title('Inspecting price effect') save and display the plot plt.savefig('C:\\Users\\Dell\\Desktop\\AV Plotting images\\matplotlib_plotting_10.png',dpi=300,bbox_inches='tight') plt.show(); pie plot plt.pie([x*100 for x in d_cuisine.values()],labels=[x for x in d_cuisine.keys()],autopct='%0.1f',explode=[0,0,0.1,0]) label the plot plt.title('Cuisine share %') plt.savefig('C:\\Users\\Dell\\Desktop\\AV Plotting images\\matplotlib_plotting_8.png',dpi=300,bbox_inches='tight') plt.show(); bar graphplt.bar(table.index,table['num_orders'])xticks plt.xticks(rotation=70) x-axis labels plt.xlabel('Food item') y-axis labels plt.ylabel('Quantity sold') plot title plt.title('Most popular food') save plot plt.savefig('C:\\Users\\Dell\\Desktop\\AV Plotting images\\matplotlib_plotting_6.png',dpi=300,bbox_inches='tight') display plot plt.show();<jupyter_code>#loading packages
import pandas as pd
import pickle
import matplotlib.pyplot as plt
plt.style.use('seaborn')
import numpy as np
import seaborn as sb
#smaller dataset
d = pd.read_pickle("../../data/dspg21RnD/smaller-final-dataset.pkl")
d["PROJECT_START_YEAR"] = d["PROJECT_START_DATE"].str[-4:]
d["PROJECT_START_YEAR"] = d.PROJECT_START_YEAR.astype(int)
d["BUDGET_START_YEAR"] = d["BUDGET_START_DATE"].str[-4:]
#d["BUDGET_START_YEAR"] = d.BUDGET_START_YEAR.astype(int)
#np.unique(d["BUDGET_START_YEAR"])
d.head()<jupyter_output><empty_output><jupyter_text>potentials for EDA:what is the avg total cost within the departments/agencies?when did most of these projects occur?what do IC_Center, ORG_Count, PI_count mean? visualization of total cost Below, are the unique department and agency names.<jupyter_code>np.unique(d["DEPARTMENT"])
np.unique(d["AGENCY"])<jupyter_output><empty_output><jupyter_text>Now, within each agency and department, I show the averages of the other variables. Specifically, we want to pay attention to the total cost,cost of the sub-projects, and the project start year.<jupyter_code>d.groupby("AGENCY").mean()
d.groupby("DEPARTMENT").mean()
d.groupby("FY").mean()<jupyter_output><empty_output><jupyter_text>most were started about 2 years before their fiscal year Next, I want to see which projects have a keyword list (project_terms), and then break that down between agencies.<jupyter_code>d["PROJECT_TERMS"].isna().sum()
d["PROJECT_TERMS"].isna().groupby(d["AGENCY"]).sum()<jupyter_output><empty_output><jupyter_text>Out of all the projects (in the smaller dataset), only 1,894 don't have the project terms. Almost all of these are within the NIH, actually. 5 are from NASA. I could make this into a bar chart of NA values but that may be a boring one, we should just report on it<jupyter_code>d.groupby("FY").count()
#fig, ax = plt.subplots()
#ax.bar(d["AGENCY"], d["PROJECT_TERMS"].isna().groupby(d["AGENCY"]).sum())
#ax.set_xlabel("Agency")
#ax.set_ylabel("Project without keyword lists")
#fig.set_figheight(10)
#fig.set_figwidth(15)
#plt.show()
d["PROJECT_ID"].groupby(d["AGENCY"]).count() # I want to see how many there are in NIH<jupyter_output><empty_output><jupyter_text>so, there are 503,298 in NIH in the smaller dataset.  If only ~2k have missing values, this seems ok.  There are also by far the most abstracts from NIH it seems, so the ones we have less for we have the project terms for all of them.<jupyter_code>d["FY_TOTAL_COST"].isna().sum()
d["FY_TOTAL_COST"].isna().groupby(d["AGENCY"]).sum()<jupyter_output><empty_output><jupyter_text>Ok so quite a lot more have missing values for the total cost.  Breaking it down by Agency all of them have some NAs, but NIH has the most by far, and again, we have the most abstracts from NIH<jupyter_code>d["FY_TOTAL_COST_SUB_PROJECTS"].isna().sum()
d["FY_TOTAL_COST_SUB_PROJECTS"].isna().groupby(d["AGENCY"]).sum()
d.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 690814 entries, 0 to 690854
Data columns (total 33 columns):
 #   Column                      Non-Null Count   Dtype  
---  ------                      --------------   -----  
 0   original index              690814 non-null  int64  
 1   PROJECT_ID                  690814 non-null  object 
 2   ABSTRACT                    690814 non-null  object 
 3   FY                          690814 non-null  object 
 4   PROJECT_TERMS               688920 non-null  object 
 5   PROJECT_TITLE               690814 non-null  object 
 6   DEPARTMENT                  690814 non-null  object 
 7   AGENCY                      690814 non-null  object 
 8   IC_CENTER                   289225 non-null  object 
 9   PROJECT_NUMBER              690814 non-null  object 
 10  PROJECT_START_DATE          690814 non-null  object 
 11  PROJECT_END_DATE            690814 non-null  object 
 12  CONTACT_PI_PROJECT_LEADER   690773 non-null  object 
 13  OTHER_PIS     [...]<jupyter_text>The variables we want to focus on for the time being are agency, department, fiscal year, project start year, budget start year, total cost, subproject cost, and which projects have a keyword list (by agency).  In total, there are 690,814 observations, and all of them have abstract data.  Of those, 688,920 have keyword lists (so, less than 2k don't have keywords). We have fiscal years and project start years for all of our abstracts.  For the total cost of the projects, we only have 473,617 values, which means that we have the cost for almost 70% of our projects. Of those projects which we have a total cost, we have the budget start date for 258,154, just over half of them (54%). We have the cost of 61,463 (9%) of subprojects.   Below, I show the distribution of projects in each fiscal year 2008-2019, followed by the distribution of project start years.<jupyter_code>fig, ax = plt.subplots()
ax.bar(d["FY"], d.groupby("FY").count())
ax.set_xlabel("Year")
ax.set_ylabel("# projects")
plt.show()
np.unique(d["PROJECT_START_YEAR"])
d.dtypes
fig, ax = plt.subplots()
ax.hist(d["PROJECT_START_YEAR"])
ax.set_xlabel("Year")
ax.set_xticklabels(d["PROJECT_START_YEAR"], rotation = 90)
ax.set_ylabel("# projects")
fig.set_figheight(10)
fig.set_figwidth(15)
plt.show()
fig, ax = plt.subplots()
ax.bar(d["BUDGET_START_YEAR"], d["FY_TOTAL_COST"])
ax.set_xlabel("Year")
ax.set_xticklabels(d["PROJECT_START_YEAR"], rotation = 90)
ax.set_ylabel("# projects")
fig.set_figheight(10)
fig.set_figwidth(15)
plt.show()<jupyter_output><empty_output><jupyter_text>d["PROJECT_START_YEAR"].mean()<jupyter_code>d["PROJECT_START_YEAR"].median()<jupyter_output><empty_output><jupyter_text>^^ why is that not going back to 1965? And why are there repeats?<jupyter_code>d["FY_TOTAL_COST"].isna().sum()
d.info()
len(d["FY_TOTAL_COST"])
690814-217197
d["PROJECT_END_DATE"].isna().sum()
fig, ax = plt.subplots()
ax.bar(d["DEPARTMENT"], d["FY_TOTAL_COST"])
plt.show()<jupyter_output><empty_output><jupyter_text>then, do the same thing with the agencies!?  Can split this up by year, also<jupyter_code>fig, ax = plt.subplots()
ax.bar(d["DEPARTMENT"], d["FY_TOTAL_COST_SUB_PROJECTS"])
plt.show()
np.unique(d["AGENCY"])
fig, ax = plt.subplots()
ax.bar(d["AGENCY"], d["FY_TOTAL_COST"])
ax.set_xticklabels(d["AGENCY"], rotation = 90)
ax.set_ylabel("Total Cost (USD)")
plt.show()
fig, ax = plt.subplots()
ax.bar(d["AGENCY"], d["FY_TOTAL_COST_SUB_PROJECTS"])
ax.set_xticklabels(d["AGENCY"], rotation = 90)
ax.set_ylabel("Total Cost (USD)")
plt.show()
d.size
d.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.130005005005005, 'fraction_numerical': 0.031156156156156155, 'mean_word_length': 4.907612712490761, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '48588', 'license': 'MIT', 'n_tokens_mistral': 2990, 'n_tokens_neox': 2754, 'n_words': 754, 'path': 'src/01_eda/01_abstract_eda.ipynb', 'repo_name': 'DSPG-Young-Scholars-Program/dspg21RnD', 'size': '8000'}