<jupyter_start><jupyter_text>![CRISP_DM](../reports/figures/CRISP_DM.png)  Business Understanding We would like to track Corona VIRUS spread across countries and with personal local informationThe general information is not so relevant for meI would like to have a deep dive local development of the spread  Goals * We would like to understand the data quality* Everything should be automated as much as possible:    how many clicks do we need to execute the full pipeline      Constraints:* Each notebook should be left clean and ready for full execution  Data Understanding * RKI (webscraping): https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/Fallzahlen.html* <NAME> (GITHUB) https://github.com/CSSEGISandData/COVID-19.git* REST API services : https://npgeo-corona-npgeo-de.hub.arcgis.com/  GITHUB csv datagit clone/pull  https://github.com/CSSEGISandData/COVID-19.git<jupyter_code>git_pull = subprocess.Popen( "git pull" ,
                          cwd = os.path.dirname('C:/Users/Alicia_Pillay/ads_covid-19/data/raw/COVID-19'),
                          shell = True,
                          stdout = subprocess.PIPE,
                          stderr = subprocess.PIPE )
(out, error) = git_pull.communicate()

print("Error :" + str(error))
print("out : " + str(out))
data_path = 'C:/Users/Alicia_Pillay/ads_covid-19/data/raw/COVID-19/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv'
pd_raw = pd.read_csv(data_path)<jupyter_output><empty_output><jupyter_text>Webscraping:<jupyter_code>page = requests.get("https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/Fallzahlen.html")
soup = BeautifulSoup(page.content, 'html.parser')
html_table = soup.find('table')
all_rows = html_table.find_all('tr')
final_data_list = []
for pos,rows in enumerate(all_rows):
    col_list=[each_col.get_text(strip=True) for each_col in rows.find_all('td')]
    final_data_list.append(col_list)
pd_daily_status=pd.DataFrame(final_data_list).dropna().rename(columns ={0:'State',
                                                                        1:'Cases',
                                                                        2:'Changes since yesterday',
                                                                        3:'Cases/100k residents',
                                                                        4:'Deaths'})
pd_daily_status.head()<jupyter_output><empty_output><jupyter_text>REST API calls<jupyter_code>data = requests.get('https://services7.arcgis.com/mOBPykOjAyBO2ZKk/arcgis/rest/services/Coronaf%C3%A4lle_in_den_Bundesl%C3%A4ndern/FeatureServer/0/query?where=1%3D1&outFields=*&outSR=4326&f=json')
json_object=json.loads(data.content)
type(json_object)
json_object.keys()
full_list=[]
for pos,each_dict in enumerate (json_object['features'][:]):
    full_list.append(each_dict['attributes'])
pd_full_list=pd.DataFrame(full_list)
pd_full_list.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11295231653703078, 'fraction_numerical': 0.013527223537368955, 'mean_word_length': 3.1370629370629373, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '632734', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 988, 'n_tokens_neox': 945, 'n_words': 161, 'path': 'notebooks/Business_Understanding-V1.0.ipynb', 'repo_name': 'Alicia021166/Enterprise-Data-Science_COVID-19', 'size': '2963'}