<jupyter_start><jupyter_text>![CRISP_DM](../reports/figures/CRISP_DM.png)  Data Preparation- focus is to undertand final data structure- support each step by visual analtics<jupyter_code>data_path='../data/raw/COVID-19/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv'
pd_raw=pd.read_csv(data_path)
pd_raw.head()
time_idx=pd_raw.columns[4:]
df_plot=pd.DataFrame({'date':time_idx})
df_plot.head()
#country_list = list(pd_raw['Country/Region'].unique())
country_list=['Italy',
              'US',
              'Spain',
              'Germany',
              'Korea, South',
              'India',
             ] 

for each in country_list:
    df_plot[each]=np.array(pd_raw[pd_raw['Country/Region']==each].iloc[:,4::].sum(axis=0))
df_plot.head()
%matplotlib inline
df_plot.set_index('date').plot()<jupyter_output><empty_output><jupyter_text>Data Type<jupyter_code>df_plot.head()
time_idx=[datetime.strptime(each,"%m/%d/%y") for each in df_plot.date]
time_str=[each.strftime('%y-%m-%d') for each in time_idx] 
df_plot.date=time_idx
type(df_plot.date[0])
df_plot.head()
df_plot.to_csv('../data/processed/COVID_small_flat_table.csv',sep=';',index=False)<jupyter_output><empty_output><jupyter_text>Relational data model - defining a primary keyIn the relational model of databases, a primary key is a specific choice of a minimal set of attributes (columns) that uniquely specify a tuple (row) in a relation (table) (source: wiki)A primary key’s main features are:- It must contain a unique value for each row of data.- It cannot contain null values.<jupyter_code>pd_raw.head()
pd_data_base=pd_raw.rename(columns={'Country/Region':'country','Province/State':'state'})
pd_data_base=pd_data_base.drop(columns={'Lat','Long'})
test_pd=pd_data_base.set_index(['state','country']).T
test_pd
test_pd=test_pd.stack(level=[0,1]).reset_index()
pd_relational_model=test_pd.rename(columns={'level_0':'date', 0:'confirmed'})
pd_relational_model.dtypes
pd_relational_model.date=pd_relational_model.date.astype('datetime64[ns]')
pd_relational_model
pd_relational_model.to_csv('../data/processed/COVID_relational_confirmed.csv',sep=';',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14949863263445762, 'fraction_numerical': 0.008204193254329991, 'mean_word_length': 6.491467576791809, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '246035', 'license': 'MIT', 'n_tokens_mistral': 812, 'n_tokens_neox': 782, 'n_words': 124, 'path': 'notebooks/Data_Preparation_V1.0.ipynb', 'repo_name': 'SUDHARSANM29/Enterprise_Data_Science_2020_Sudharsan_Mukundan_415774', 'size': '2194'}