<jupyter_start><jupyter_text>Contract Awards by  the Centers for Medicare and Medicaid Services (CMS)The dataset comes from https://www.usaspending.gov//download_center/custom_award_data using filters to select all Prime Awards types, Agency = Department of Health and Human Services, Sub Agency = Centers for Medicare and Medicaid Services, by All Recipient Location, by Action Date, for FY 2019.  Data on Health and Human Services (HHS) CMS' contract awards for fiscal year (FY) 2019.In the spirit of transparency and full disclosure, the www.usaspending.gov site includes information about how federal dollars have been spent. Tools on the site allow members of the public to access data sets for their own evaluation or to filter and visualize data online. I am interested in seeing statistics about the contract awards in FY 2019, especially to see which contract award types garner the highest dollar awards, which types of work have the most number of proposals received (and which ones do not receive multiple proposals), and which award recipients came away with the biggest wins.<jupyter_code>import os
os.getcwd() # gives the path of the current working directory (cwd)

import numpy as np
import pandas as pd
#contracts = pd.read_csv('C:\\Users\\Jennifer\\Downloads\\FY2019HHSContractAwards.csv') # This reads in my csv file.
contracts = pd.read_csv('/FY2019HHSContractAwards.csv')
contracts.head() # This lists the first 5 rows of the dataframe with all of the columns (276).
contracts.shape<jupyter_output><empty_output><jupyter_text>My dataframe has 3,787 rows with 276 columns.<jupyter_code># This was a first attempt to subset the dataframe but this isn't what I really wanted.
# Implicit index with iloc #before comma is the rows and after comma is columns, with the ends excluded.
# df1.iloc[:2,:1 ]

#contracts_short = contracts.iloc[:,:6] #this pulls all the rows but only the first 5 columns. 
#I want to skip around and pull different columns.
#contracts_short.head()
#dataframe.filter([items,like,regex,axis])
# This filter command makes a subset of the dataframe with only the 19 variables/columns that I am interested in. 

contracts_short = contracts.filter(['contract_transaction_unique_key','parent_award_agency_name','total_dollars_obligated','base_and_exercised_options_value','action_date','period_of_performance_start_date','funding_sub_agency_name','recipient_name','recipient_doing_business_as_name','recipient_state_code','type_of_contract_pricing','product_or_service_code_description','naics_description','solicitation_procedures','type_of_set_aside','number_of_offers_received','multi_year_contract','veteran_owned_business','service_disabled_veteran_owned_business'])
contracts_short.head() # typed the individual column names to pull these into the smaller df called contracts_short
contracts_short.shape # This gives me the number of (rows, columns) in my shortened dataframe.
contracts_short.dtypes # This gives the data type for each variable (column).<jupyter_output><empty_output><jupyter_text>The subset dataframe has different kinds of variables. The dataframe has strings, boolean (y/n and t/f), numeric, and date values.<jupyter_code>contracts_short.describe() # Describe gives the stats for the numeric columns.
contracts_short.describe(include="all") # This gives stats/counts on the non-numeric column types, too.
print(contracts_short) #this shows me the first and last 5 rows
type(contracts_short)<jupyter_output>contract_transaction_unique_key  ... service_disabled_veteran_owned_business
0     7530_7530_HHSM500T0016_P00004_HHSM500201400034I_0  ...                                       f
1     7530_7530_HHSM500T0011_P00007_HHSM500200400007I_0  ...                                       f
2     7530_7530_HHSM500T0016_P00006_HHSM500201400034I_0  ...                                       f
3       7530_4732_HHSM500201700031G_P00006_GS35F328DA_0  ...                                       f
4         7530_7530_75FCMC19F0002_0_HHSM500201600051I_0  ...                                       f
...                                                 ...  ...                                     ...
3782        7530_4730_75FCMC18F0065_P00001_GS07F9186S_0  ...                                       f
3783      7530_-NONE-_HHSM500201700020P_P00001_-NONE-_0  ...                                       f
3784               7530_-NONE-_75FCMC19C0004_0_-NONE-_0  ...                               [...]<jupyter_text>My variable total_dollars_obligated does not have any na's but just to be sure I will remove them.<jupyter_code>#data_na = (data.dropna(subset=['Tax Returns']))
contracts_short_na = (contracts_short.dropna(subset=['total_dollars_obligated'])) #this removes the records that have na's in this variable.
print(contracts_short_na)
type(contracts_short_na)<jupyter_output>contract_transaction_unique_key  ... service_disabled_veteran_owned_business
0     7530_7530_HHSM500T0016_P00004_HHSM500201400034I_0  ...                                       f
1     7530_7530_HHSM500T0011_P00007_HHSM500200400007I_0  ...                                       f
2     7530_7530_HHSM500T0016_P00006_HHSM500201400034I_0  ...                                       f
3       7530_4732_HHSM500201700031G_P00006_GS35F328DA_0  ...                                       f
4         7530_7530_75FCMC19F0002_0_HHSM500201600051I_0  ...                                       f
...                                                 ...  ...                                     ...
3782        7530_4730_75FCMC18F0065_P00001_GS07F9186S_0  ...                                       f
3783      7530_-NONE-_HHSM500201700020P_P00001_-NONE-_0  ...                                       f
3784               7530_-NONE-_75FCMC19C0004_0_-NONE-_0  ...                               [...]<jupyter_text>Since my number of rows did not change (3787) I know that no records were removed due to na's.  I will continue with some exploratory data analysis (EDA) using matplotlib.<jupyter_code># This didn't work out quite as planned so i've abandoned this method.
#import matplotlib as mpl
#import matplotlib.pyplot as plt
#x = np.random.randn(1000)
#plt.hist(x); # or 
#plt.hist(x, bins=30,normed=True,alpha=0.5,histtype='stpfilled',color='steelblue',edgecolor='none');

#offers = contracts_short.filter(['number_of_offers_received'])
#offers = offers.fillna(0)
#offers = contracts_short.pd('number_of_offers_received')
#plt.hist(offers)
#plt.show()<jupyter_output><empty_output><jupyter_text>This will give me a histogram of the number of offers for each contract.<jupyter_code>import matplotlib as mpl
import matplotlib.pyplot as plt
_ = plt.hist(contracts_short.number_of_offers_received)
_ = plt.xlabel('Number of Offers sent to CMS for each Contract FY19')
_ = plt.ylabel('Number of Contracts')<jupyter_output>/usr/local/lib/python3.6/dist-packages/numpy/lib/histograms.py:839: RuntimeWarning: invalid value encountered in greater_equal
  keep = (tmp_a >= first_edge)
/usr/local/lib/python3.6/dist-packages/numpy/lib/histograms.py:840: RuntimeWarning: invalid value encountered in less_equal
  keep &= (tmp_a <= last_edge)<jupyter_text>I will use seaborn styling to show this histogram again.<jupyter_code>import seaborn as sns
sns.set()
_ = plt.hist(contracts_short.number_of_offers_received)
_ = plt.xlabel('Number of Offers sent to CMS for each Contract FY19')
_ = plt.ylabel('Number of Contracts')<jupyter_output>/usr/local/lib/python3.6/dist-packages/numpy/lib/histograms.py:839: RuntimeWarning: invalid value encountered in greater_equal
  keep = (tmp_a >= first_edge)
/usr/local/lib/python3.6/dist-packages/numpy/lib/histograms.py:840: RuntimeWarning: invalid value encountered in less_equal
  keep &= (tmp_a <= last_edge)<jupyter_text>I will make the histogram again but change the bin number.<jupyter_code>_ = plt.hist(contracts_short.number_of_offers_received, bins=100)
_ = plt.xlabel('Number of Offers sent to CMS for each Contract FY19')
_ = plt.ylabel('Number of Contracts')<jupyter_output>/usr/local/lib/python3.6/dist-packages/numpy/lib/histograms.py:839: RuntimeWarning: invalid value encountered in greater_equal
  keep = (tmp_a >= first_edge)
/usr/local/lib/python3.6/dist-packages/numpy/lib/histograms.py:840: RuntimeWarning: invalid value encountered in less_equal
  keep &= (tmp_a <= last_edge)<jupyter_text>This will give me a histogram of the Total Dollars Obligated for the fiscal year.<jupyter_code>plt.hist(contracts_short.total_dollars_obligated)
plt.xlabel('Total Dollars Obligated FY19')
plt.ylabel('Number of Contracts')<jupyter_output><empty_output><jupyter_text>This will give me a histogram of the Total Dollars for the Base Year and already Exercised Options.<jupyter_code>plt.hist(contracts_short.base_and_exercised_options_value)
plt.xlabel('Total Dollars - Base Year and Exercised Options')
plt.ylabel('Number of Contracts')<jupyter_output>/usr/local/lib/python3.6/dist-packages/numpy/lib/histograms.py:839: RuntimeWarning: invalid value encountered in greater_equal
  keep = (tmp_a >= first_edge)
/usr/local/lib/python3.6/dist-packages/numpy/lib/histograms.py:840: RuntimeWarning: invalid value encountered in less_equal
  keep &= (tmp_a <= last_edge)<jupyter_text>Going back to the Total Dollars Obligated for the FY19, I will try different bin values to make the histogram reveal more about the data.<jupyter_code>sns.set()
plt.hist(contracts_short.total_dollars_obligated,bins=50)
plt.xlabel('Total Dollars Obligated FY19')
plt.ylabel('Number of Contracts')<jupyter_output><empty_output><jupyter_text>BootstrappingI will need to pull just the one variable out. I will need to take random samples of my records for the variable_total_dollars obligated and append the samples onto the data.<jupyter_code>#mdtaxrt = data_na['Tax Returns']
#type(mdtaxrt)
obligated = contracts_short['total_dollars_obligated'] # this keeps just the one variable.
print(obligated)
type(obligated)
data_sample = np.random.choice(obligated,size = 100, replace = False) # this is just a random sample of 100 records
np.mean(data_sample) # the mean of my one sample
# replace = False means the values are removed from the dataset as the function picks the random sample
np.median(data_sample)# the median of my one sample
np.std(data_sample) # the standard deviation of my one sample
from sklearn.utils import resample #we use the resample 
n_iterations = 2000

boot_samples = list()
mean_boot_samples = list()
median_boot_samples = list()

for i in range(n_iterations): #this does the bootstrapping
    boot_sample = resample(data_sample)
    boot_samples.append(boot_sample) # this appends the new sample onto the rest
    
    mean_boot_samples.append(np.mean(boot_sample)) #calculates the mean of the sample and appends it to the list
    
    median_boot_samples.append(np.median(boot_sample)) #calculates the median of the sample and appends it to the list
    
print("The Length of the boot_samples is: " + str(len(boot_samples)))
print("The mean of the bootstrapped data is: " + str(np.mean(mean_boot_samples)))
print("The median of the bootstrapped data is: " + str(np.median(median_boot_samples)))

np.mean(obligated)
print("The mean of the full dataset is: " + str(np.mean(obligated)))<jupyter_output>The mean of the full dataset is: 51740344.17200949<jupyter_text>Confidence Interval for the Median<jupyter_code>resamples_medians = pd.Series(index=np.arange(1000))
for i in range(1000): 
              resamples_medians[i] = np.median(resample(data_sample))

print(resamples_medians.median())
print(resamples_medians.mean())
print(resamples_medians.quantile(0.025))
print(resamples_medians.quantile(0.975))
np.median((obligated))
print("the median of the full dataset is: " + str(np.median(obligated)))<jupyter_output>the median of the full dataset is: 4080376.73
{'fraction_non_alphanumeric': 0.0837917900973339, 'fraction_numerical': 0.0543377063055438, 'mean_word_length': 3.92743953294412, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 40, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '287264', 'license': 'Apache-2.0', 'n_tokens_mistral': 4111, 'n_tokens_neox': 3589, 'n_words': 993, 'path': '20200414 Project1JParaboschi.ipynb', 'repo_name': 'jenniferparaboschi/MCData110Fall2019', 'size': '11815'}