<jupyter_start><jupyter_text>**NB:** This notebook was created using **Google Colab** [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/A2-insights/Covid19-Data-Insights/blob/master/notebooks/A2_insights_Lab_Covid19_EDA.ipynb)  **$A^{2}$ Insights Lab** *presents*The COVID-19 informative analysis crafted using python, pandas and other modern technologies. --- **Disclaimer**The information contained in this Notebook is for **Educational purposes only**. The Data Analysis and code is provided by **$A^{2}$ Insights Lab**, Conclusions or point of views made from this notebook should be interpreted as Opinoins and not as health professional advice. **Any reliance you place on such information is therefore strictly at your own risk.****We Do not Claim** ownership nor correctness of the data used, therefore In no event will we be liable for any violation of the government Regulations or damage including without limitation, indirect or consequential loss or damage whatsoever arising from loss of data or misleading information, or in connection with, the use of this information.  **Imports**<jupyter_code># Data manipulation 
import pandas as pd 
import numpy as np
import requests

#  Data Visualization
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

# Extas
import warnings

# Disabling library Warnings
warnings.filterwarnings('ignore')

# Out Put Settings, precision
pd.set_option('precision', 3)<jupyter_output><empty_output><jupyter_text>**Loading Data** ---**API starter:**>Use this notebook to connect to the COVID-19 database for access to our datasources.>Use [these API's](https://ijvcpr9af1.execute-api.eu-west-1.amazonaws.com/api/) to access the data.--- Build a dataframe using the following code:```url = "INSERT API URL"``` ```headers = {'x-api-key': "INSERT API KEY HERE"}``````response = requests.request("GET", url, headers=headers)``````x = response.json()``````df = pd.DataFrame(x)``` **Call the following function to read the data To a Dataframe**```load_covid19(url, API_KEY)```  CONSTANTS<jupyter_code># CONSTANTS
API_KEY = '<KEY>'

# External Data Sources URLs
SYMPTOMS_URL = 'https://en.wikipedia.org/wiki/Coronavirus'

# FULL GLOBAL DATASETS
GLOBAL_URL = 'https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/ecdc/full_data.csv'
LOCATIONS_URL = 'https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/ecdc/locations.csv'

# GLOBAL DATA TO BE MERGED
CONFIRMED_URL = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv'
RECOVERED_URL = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_recovered_global.csv'
DEATHS_URL = 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv'

# SOUTH AFRICA DATASETS
RSA_URL =  'https://raw.githubusercontent.com/dsfsi/covid19za/master/data/covid19za_provincial_cumulative_timeline_confirmed.csv'

# DISTRICT DATA
WESTERN_CAPE_URL = 'https://raw.githubusercontent.com/dsfsi/covid19za/master/data/district_data/provincial_wc_cumulative.csv'
GAUTENG_URL = 'https://raw.githubusercontent.com/dsfsi/covid19za/master/data/district_data/provincial_gp_cumulative.csv'


# API URLs
url_CasesGlobalView = 'https://9gnht4xyvf.execute-api.eu-west-1.amazonaws.com/api/get_table/CasesGlobalView'
url_CasesLocalView  = 'https://9gnht4xyvf.execute-api.eu-west-1.amazonaws.com/api/get_table/CasesLocalView'
ulr_CounterMeasureView = 'https://9gnht4xyvf.execute-api.eu-west-1.amazonaws.com/api/get_table/CounterMeasureView'<jupyter_output><empty_output><jupyter_text>**GATHER EXTERNAL DATA**---Functions to gather data from other sources[link text](https://)<jupyter_code>def scrape_covid19_symptoms(url):
    """scrape_covid19_symptoms(url)
    
    Scrapes covid 19 symptoms from wikipedia coronavirus pages
    
    PARAMETERS: 
        url (string): wikipedia coronavirus url page
    """

    # get all tables and extract covid 19 symptoms table
    df = pd.read_html(url, header=None)
    df = df[1][11:17].loc[: , ['Unnamed: 0', 'SARS-CoV-2'] ].reset_index().drop('index', axis=1)
    df.rename(columns={'Unnamed: 0': 'symptoms', 'SARS-CoV-2': 'covid19_percentage'}, inplace= True)
    
    return df
def load_covid19(url, API_KEY = '<KEY>' ):
  """load_covid19(url)

  DESCRIPTION:
        The function reads in an API URL for COVID-19 Data and returns a pandas DataFrame.

  PARAMETERS:
        url (string): API URL.

  RETURNS:
      (DataFrame): Data in a form of a pandas dataframe.
  
  """

  headers = {
    'x-api-key': API_KEY
    }
  response = requests.request("GET", url, headers=headers)
  x = response.json()

  return pd.DataFrame(x)<jupyter_output><empty_output><jupyter_text>**Load Data to DataFrames**---Calling functions to lead data<jupyter_code># Reading Data to DataFrames
global_view = load_covid19(url_CasesGlobalView)
local_view = load_covid19(url_CasesLocalView)
counterMeasure_view = load_covid19(ulr_CounterMeasureView)

# loading covid 19 symptoms dataframe
symptoms_df = scrape_covid19_symptoms(SYMPTOMS_URL)

# loading external sources
global_confirmed_ = pd.read_csv(CONFIRMED_URL)
global_recorvered_ =  pd.read_csv(RECOVERED_URL)
global_deaths_ =  pd.read_csv(DEATHS_URL)<jupyter_output><empty_output><jupyter_text>**Domain Knowledge & Few Data Checks** *Quick Descriptive Analysis, Data Values & Data Types*---Lets get to understand the **Features** and **Categorical** values.- The **cardinality of Categorical Features** might not change as the data increases.- **investigate Categorical Features :**  - country  - province/state/region  - measures  - and more if any---- **investigate the Data type per Feature:**  - **Date** (format)  - **floats** stored as **objects** or vice versa  - **Down cast data types** to **pandas data types** which **reduce memory usage**.---- **Investigate Feature Values:**  - **Missing Data values**    - missing at Random    - missing by aggregation    - indicated as missing but the actual value is Zero. - **non-atomic Features & format:**    - Create a new Feature    - Remove part of the data    - Use **iso_** features to correct cuntries.---   **CasesGlobalView**<jupyter_code># Dimension of the global_view data
print(f'The global  Instances are : {global_view.shape[0]}')
print(f'The global  Features are  : {global_view.shape[1]}')
# view 5 samples of the data
global_view.sample(5)
# understanding global view columns
global_cols = global_view.columns.to_list()
global_cols
# data types per Feature, Memeory size
global_view.info()
# get count for null values
global_view.isnull().mean()<jupyter_output><empty_output><jupyter_text>**Notes 2** Missing Data:> We need to understand the Nature of missing values before we can drop the records or impute the values. > NB: Imputaion / droping records with missing values may yeild misleading results and may affect the interpretation of the pandemic outbreak. The following columns have more less **10%** missing values out of current loaded observations :* confirmed_daily                    * deaths_daily                   * recovered_daily                 * daily_change_in_active_cases     * active_dailiy_growth_rate       * active_rolling_3_day_growth_rate<jupyter_code># Eliminate the Logitude and latitude the view the summary statistics
global_view.drop(['lat','long'], axis=1).describe()<jupyter_output><empty_output><jupyter_text>**Notes 3**> Understanding the **mean**, **std** and **MAx** for the following features/ Variables:- **confirmed**- **deaths** 	- **recovered**> **Ideal Scenario*** The **std** and **mean** for **confirmed** and **deaths** are expected  to have a very high variation, this will mean that the virus **preading-rate** and **death-rate** are droping(flaterning the  curve).* The **std** and **mean** for **Recovered** are expected to have a less to no variation and this will mean that more people are recovering at a short period of time.[[1]('')]<jupyter_code># get number of countries 
global_view['country'].nunique()<jupyter_output><empty_output><jupyter_text>**notes **we got **185 countries**, this should be considered when ploting vs country.  **CasesLocalView**<jupyter_code># Dimension of the global_view data
print(f'The local  Instances are : {local_view.shape[0]}')
print(f'The local  Features are  : {local_view.shape[1]}')
# view 5 samples of the local view data
local_view.sample(5)
# data types
local_view.info()
# get count for Missing Data
local_view.isnull().mean()
# Remove the id & country_id Features since they are not informative on the summary statistics
local_view.drop(['id','country_id'], axis=1).describe()
# group by location and get the size 
local_view.groupby('location').size()<jupyter_output><empty_output><jupyter_text>**CounterMeasureView**<jupyter_code># Dimension of the global_view data
print(f'The Counter Measure  Instances are : {counterMeasure_view.shape[0]}')
print(f'The Counter Measure  Features are  : {counterMeasure_view.shape[1]}')

# view 5 samples of the Counter Measure View data
counterMeasure_view.sample(5)
# data type per  Feature and memory size
counterMeasure_view.info()
# cheget the % of missing values
counterMeasure_view.isnull().mean()
# Summary statisctics
counterMeasure_view.describe()
# investigate the number of measures
counterMeasure_view['measure'].nunique()<jupyter_output><empty_output><jupyter_text>**Notes **we got **23 measures**,So need to be careful when using scatter plots and not cluster by measures.  **Symptoms Data**<jupyter_code># Dimension of the symptoms_df data
print(f'The Symptoms  Instances are : {symptoms_df.shape[0]}')
print(f'The Symptoms Features are  : {symptoms_df.shape[1]}')
# view data samples
symptoms_df<jupyter_output><empty_output><jupyter_text>clean up the % and extra numbers<jupyter_code># data types 
symptoms_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 6 entries, 0 to 5
Data columns (total 2 columns):
symptoms              6 non-null object
covid19_percentage    6 non-null object
dtypes: object(2)
memory usage: 224.0+ bytes<jupyter_text>cast **percentage** to be a float not object  **Data Preprocessing***   Cast the DATATYPES to reduce memory usage*   Create pipeline To clean the Data*   Pass all the data Stream through the pipeline   **LOGIC**---custorm Functions to preprocess all the data<jupyter_code>def preprocess_symptoms_df(df):

    # preprocening the dataframe
    df.replace(regex=['%\[69]', '%\[71]'], value='', inplace=True)
    
    # cast object to float and reduce memory usage
    df.covid19_percentage = df.covid19_percentage.astype('float')

    return df<jupyter_output><empty_output><jupyter_text>**PREPROCESING** ---<jupyter_code>symptoms_df = preprocess_symptoms_df(symptoms_df)
symptoms_df<jupyter_output><empty_output><jupyter_text>**Visualizations***Lets plan the charts so as to better group them based on the information then convey.*---**Groups of Plots:**1.   Summary plots e.g pie charts2.   Trends plots (Time series)2.   Distribution plots3.   Relationship plots---1.  **Trends:**- **investigate by date of report** (chart Type = ?) time series    - Total deaths, recovered and confirmed vs date of report    - daily deaths, recovered and confirmed vs date of report---2.   **Distribution plots:**- **investigate by country**     - deaths vs country     - recovered vs country     - confirmed vs country- **investigate by province**- **investigate by measures** ---<jupyter_code>## plotting the symptoms VS covid19_percentage
plt.title('plotting the symptoms VS covid19_percentage')
sns.barplot(x=symptoms_df['symptoms'],y=symptoms_df['covid19_percentage']);
plt.xticks(rotation=60)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11529431494214322, 'fraction_numerical': 0.013499916149589133, 'mean_word_length': 5.374665954035275, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 65, 'https://': 16, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '52017', 'license': 'MIT', 'n_tokens_mistral': 3940, 'n_tokens_neox': 3711, 'n_words': 1058, 'path': 'notebooks/A2_insights_Lab_Covid19_EDA_v01.ipynb', 'repo_name': 'A2-insights/Covid19-Data-Insights', 'size': '11996'}