<jupyter_start><jupyter_text>`It is worth noting that the user, who want to use this notebook to process the data from Spanish to English, does not need to understand these codes but run the entire notebook directly will be enough`  The newest "COVID19_MEXICO.csv" file  can be installed from: https://www.gob.mx/salud/documentos/datos-abiertos-152127As one column in the original database was written in UTF-8 whereas other was written in a different format which made impossible to read the dataset directly, consequently it was necessary to solve this problem by the following Python codes to convert all datas in the same format and also translate to English.We will need the following function in order to transform the column 'MUNICIPAL_RES' which contains more than 2500 municipalities which were encoded by numbers.To use the following function you will need the file **Municipals_Catalog.csv**, whose name ('**Municipals_Catalog.csv**') was **copied and pasted separately by author** from the original datasheet which called "**Catálogo MUNICIPIOS**". This file can be found in: **dictionary_data_COVID19 >> Catalogos_0412.csv** in the link provided from the outset.<jupyter_code>import csv 

Dic_Entity = {1:'AGUASCALIENTES',2:'BAJA CALIFORNIA',3:'BAJA CALIFORNIA SUR',4:'CAMPECHE',5:'COAHUILA DE ZARAGOZA',
              6:'COLIMA',7:'CHIAPAS',8:'CHIHUAHUA',9:'MEXICO CITY',10:'DURANGO',11:'GUANAJUATO',12:'GUERRERO',
              13:'HIDALGO',14:'JALISCO',15:'STATE OF MEXICO',16:'MICHOACÁN DE OCAMPO',17:'MORELOS',18:'NAYARIT',19:'NUEVO LEÓN',
              20:'OAXACA',21:'PUEBLA',22:'QUERÉTARO',23:'QUINTANA ROO',24:'SAN LUIS POTOSÍ',25:'SINALOA',26:'SONORA',
              27:'TABASCO',28:'TAMAULIPAS',29:'TLAXCALA',30:'VERACRUZ',31:'YUCATÁN',32:'ZACATECAS',
              36:'UNITED MEXICAN STATES',97:'NOT APLIED',98:'IGNORED',99:float('nan')}

def Transfor_Mun_Res(filename = 'Municipals_Catalog.csv', Dic_Entity = Dic_Entity):
    """
    Return a dictionary where the key is the Entity name and the value is another subdictionary; 
    the key in subdictionary is the Municipal number and the value is the Municipal Name
    """
    dic = {}
    with open(filename, encoding='utf-8') as csvfile:
        reader = csv.reader(csvfile, delimiter=';')
        next(reader) # we skip the header
        
        for line in reader:
            Municipal_Code, Municipal, Entity_Code = line
            
            if Entity_Code < '10':                 # if the Entity_Code is '0X' where X is a number in [1-9]
                Entity_Code = int(Entity_Code[-1]) # transform '0X' into an integer X
        
            else:                              # If the Entity_Code is already a number > 9
                Entity_Code = int(Entity_Code) # Transform directly 'XX' into an integer XX / X is a natural number in [1-9]
            
            dic_value = dic.setdefault(Entity_Code, dict())
            dic_value[int(Municipal_Code)] = Municipal
            
    for elem in dic:
        dic[elem][999] = float('nan')          # The value 99 or 999 mean Unspecified, which we will consider as NaN value 
        
    return dic
Dic_Municipals = Transfor_Mun_Res() #We will need this dictionary in the next block<jupyter_output><empty_output><jupyter_text>Reformatting the column **'MUNICIPAL_RES'**, **'COUNTRY_NATIONALITY'** and **'COUNTRY_ORIGIN'** due to the convenience, with them we then create a new csv file.<jupyter_code>import csv     # Import csv library

### Create the Dictionary which contains the pair (key, value) where the key is the original format and 
### the value is the correspondent transformation word in English.

Dic = {'ArchipiÃ©lago de Svalbard':'Svalbard', 'AscensiÃ³n':'Ascension Island', 'AzerbaiyÃ¡n - Islas Azores':'Azores',
      'BÃ©lgica':'Belgium', 'CamerÃºn':'kamerun', 'CanadÃ¡':'Canada', 'EspaÃ±a':'Spain', 
      'Estados Unidos de AmÃ©rica':'United States', 'Gran BretaÃ±a (Reino Unido)':'Great Britain',
      'HaitÃ­':'Haiti', 'HungrÃ­a':'Hungary', 'IrÃ¡n':'Iran', 
      'Islas VÃ­rgenes de Estados Unidos de AmÃ©rica':'United States Virgin Islands', 'JapÃ³n':'Japan',
      'LÃ­bano':'Lebanon','MÃ©xico':'Mexico','PakistÃ¡n':'Pakistan','PerÃº':'Peru','Principado de MÃ³naco':'Monaco',
      'RepÃºblica Checa y RepÃºblica Eslovaca':'Czech Republic and Slovak Republic', 
      'RepÃºblica de Angola': 'Angola', 'RepÃºblica de Corea':'South Korea', 'RepÃºblica de Costa Rica':'Costa Rica',
      'RepÃºblica de Guyana':'Guyana', 'RepÃºblica de Honduras':'Honduras', 'RepÃºblica de Mauricio':'Mauritius',
      'RepÃºblica de PanamÃ¡':'Panama','RepÃºblica de Trinidad y Tobago':'Trinidad and Tobago',
      'RepÃºblica DemocrÃ¡tica de Corea':'North Korea', 'RepÃºblica Dominicana':'Dominican Republic',
      'RepÃºblica Oriental del Uruguay':'Uruguay', 'Republica DemocrÃ¡tica del Congo':'Congo', 'SudÃ¡frica':'South Africa',
      'TaiwÃ¡n':'Taiwan', 'TurquÃ­a':'Turkey',
        'Alemania':'German', 'Archipiélago de Svalbard':'Svalbard','Argelia':'Algeria', 'Argentina':'Argentina', 
        'Ascensión':'Ascension Island', 'Australia':'Australia', 'Austria':'Austria',
        'Azerbaiyán - Islas Azores':'Azores', 'Bangladesh':'Bangladesh', 'Belice':'Belice', 'Bolivia':'Bolivia',
        'Bosnia y Herzegovina':'Bosnia and Herzegovina', 'Brasil':'Brazil', 'Bélgica':'Gallia Belgica', 
        'Camerún':'Cameroon', 'Canadá':'Canada', 'Chile':'Chile', 'China':'China', 'Colombia':'Columbia',
        'Commonwealth de Dominica':'Dominica', 'Congo':'Congo', 'Costa de Marfil':'Ivory Coast', 'Croacia':'Croatia',
        'Cuba':'Cuba', 'Dinamarca':'Denmark', 'Ecuador':'Ecuador', 'Egipto':'Egypt', 'El Salvador':'El Salvador',
        'Eritrea':'Eritrea', 'Eslovaquia':'Eslovaquia', 'Eslovenia':'Slovenia', 'España':'Spain',
        'Estados Unidos de América':'United States', 'Filipinas':'Philippines', 'Finlandia':'Filandia',
        'Francia':'France', 'Ghana':'Ghana', 'Gran Bretaña (Reino Unido)':'Great Britain', 'Grecia':'Greece', 
        'Guatemala':'Guatemala', 'Guinea Ecuatorial':'Equatorial Guinea', 'Haití':'Haiti', 'Holanda':'Holland', 
        'Hong Kong':'Hong Kong', 'Hungría':'Hungary', 'India':'India', 'Indonesia':'Indonesia',
        'Irlanda':'Ireland', 'Irán':'Iran', 'Islandia':'Iceland',
        'Islas Vírgenes de Estados Unidos de América':'United States Virgin Islands', 'Israel':'Israel',
        'Italia':'Italy', 'Jamaica':'Jamaica', 'Japón':'Japan', 'Letonia':'Latvia', 'Libia':'Libya', 
        'Lituania':'Lithuania', 'Líbano':'Lebanon', 'Macao':'Macao', 'Malasia':'Malaysia', 'Marruecos':'Morocco',
        'Micronesia':'Micronesia', 'Moldavia':'Moldova', 'México':'Mexico','Nepal':'Nepal', 'Nicaragua':'Nicaragua',
        'Noruega':'Norway', 'Nueva Zelandia':'New Zealand', 'Otro':'Other', 'Pakistán':'Pakistan', 'Paraguay':'Paraguay',
        'Perú':'Peru', 'Polonia':'Poland', 'Portugal':'Portugal', 'Principado de Mónaco':'Monaco', 
        'Republica Democrática del Congo':'Congo', 'República Checa y República Eslovaca':'Czech Republic and Slovak Republic',
        'República Democrática de Corea':'North Korea', 'República Dominicana':'Dominican Republic',
        'República Oriental del Uruguay':'Uruguay', 'República de Angola':'Angola', 'República de Corea':'South Korea',
        'República de Costa Rica':'Costa Rica', 'República de Guyana':'Guyana', 'República de Honduras':'Honduras',
        'República de Mauricio':'Mauritius', 'República de Panamá':'Panama',
        'República de Trinidad y Tobago':'Trinidad and Tobago', 'Rumania':'Romania', 'Rusia':'Russia', 
        'SE DESCONOCE':'Unknown', 'Siria':'Syria', 'Sudáfrica':'South Africa', 'Suecia':'Sweden', 'Suiza':'Switzerland',
        'Taiwán':'Taiwan', 'Trieste':'Trieste', 'Turquía':'Turkey', 'Ucrania':'Ukraine', 'Venezuela':'Venezuela', 
        'Vietnam':'Vietnam', 'Zimbabwe':'Zimbabwe', 'Zona Neutral':'Demilitarized zone', '99':float('nan')}


def formatting_countries_entities_municipals(filename, dic = Dic, dic_Municipals = Dic_Municipals):
    
    new_csvfile = 'Trans_COVID19_MEXICO.csv'
    with open(new_csvfile,'w', encoding='utf-8') as new_csvfile:
        writer = csv.writer(new_csvfile, delimiter=',', lineterminator = '\n') #lineterminator to avoid skipping lines when writing to csv
        # write the header:
        writer.writerow(["LAST_UPDATE",             # The last update date about the information regarding the patient
                             "ID_REGISTRATION",     # ID Number of the patient
                             "ORIGIN",              # Whether Sentinel surveillance was carried out through the respiratory disease monitoring health unit system (USMER) or not.
                             "SECTOR",
                             "ENTITY_UM",
                             "SEX",
                             "ENTITY_NAT",         
                             "ENTITY_RES",
                             "MUNICIPALITY_RES",
                             "PATIENT_TYPE",       # Whether the patient was hospitalized after being diagnosticated
                             "ADMISSION_DATE", 
                             "SYMPTOMS_DATE",
                             "DEATH_DATE",
                             "INTUBATED",
                             "PNEUMONIA",
                             "AGE",
                             "NATIONALITY",         # Whether the patient is Mexican or Foreigner
                             "PREGNANT",
                             "SPEAK_INDIGENOUS_LANGUAGE",
                             "INDIGENOUS",          # Indigenous patient
                             "DIABETES",
                             "COPD",                # Chronic obstructive pulmonary disease
                             "ASTHMA",           
                             "INMUSUPR",            # Immunosuppression
                             "HYPERTENSION",
                             "OTHER_DISEASE",
                             "CARDIOVASCULAR",
                             "OBESITY",
                             "CHRONIC_KIDNEY_DISEASE",
                             "SMOKE",
                             "OTHER_CASE_CONTACT",  # Whether the patient contacted with other diagnosticated case with SARS CoV-2
                             "TESTED",              # Whether the patient were took a sample to test coronavirus possession
                             "RESULT_LAB",          # Positivo SARS-CoV-2 or not (Result of laboratory)
                             "FINAL_CLASIFICATION", #
                             "MIGRANT",
                             "COUNTRY_NATIONALITY", # The nationality of the foreigners
                             "COUNTRY_ORIGIN",      # The via where the infected patients came from
                             "UCI"])
        
        with open(filename) as csvfile:
            reader = csv.reader(csvfile, delimiter=',')
            
            next(reader) # skip the heading
            for line in reader:
                ### Reformatting the wrong string which was written in Spanish
                line[-3] = dic[line[-3]] if line[-3] in dic else 'Other'    # Reformatting the Nationality
                line[-2] = dic[line[-2]] if line[-2] in dic else 'Other'    # Reformatting the Origin Country
                try:
                    Municipals = dic_Municipals[int(line[7])] #Find the municipals which a determinated Entity contains
                    line[8] = Municipals[int(line[8])]
                except KeyError: 
                    # However, there are a tiny portion of rows which represent incoherence between Entity_Res and Municipal_Res, 
                    # probably due to artificial error, then we decided to assign those small proportion into NaN value
                    line[8] = float('nan')
                
                ### Reformatting all Date column, 9999-99-99 is considered as NaN value in the original Date Variable
                line[0] = float('nan') if line[0] == '9999-99-99' else line[0]
                line[10] = float('nan') if line[10] == '9999-99-99' else line[10]
                line[11] = float('nan') if line[11] == '9999-99-99' else line[11]
                line[12] = float('nan') if line[12] == '9999-99-99' else line[12]
                
                ### Write the lines in csv file with corrected version
                writer.writerow(line)<jupyter_output><empty_output><jupyter_text>Reformatting the datasheet into one new csvfile ''Trans_COVID19_MEXICO.csv"<jupyter_code>formatting_countries_entities_municipals("201102COVID19MEXICO.csv", Dic) # Transformation<jupyter_output><empty_output><jupyter_text>With the new csv file generated before, we now refformate the rest of columns which can be done easily and efficiently by the function **map()**, given a python **dictionary** (see the following chunks).<jupyter_code>import pandas as pd
df = pd.read_csv("Trans_COVID19_MEXICO.csv", delimiter = ',') # Read the new datasheet, wholly coded with UTF-8
df  # Browse the generated dataframe<jupyter_output><empty_output><jupyter_text>As we can observe, the majority of variables were encoded with numbers, in order in improve the visualization we then transform them to the original value and translate them in English(see **"dictionary_data_COVID19"** for further information)<jupyter_code># Reformatting the variable Origin:
Dic_Origin = {1:'USMER', 2:'NOT USMER', 99:float('nan')} # The value 99 means Unspecified, which we will consider as NaN value
df['ORIGIN'] = df['ORIGIN'].map(Dic_Origin)
print(df.ORIGIN.value_counts()) # A brief information
# Reformatting the variable Sector. Note: Some of them are agency from Mexico and are writtin in abbreviation
Dic_Sector = {1:'RED CROSS',  2:'DIF', 3:'STATE', 4:'IMSS', 5:'IMSS-BIENESTAR', 6:'ISSSTE', 7:'MUNICIPAL', 
              8:'PEMEX', 9:'PRIVATE', 10:'SEDENA', 11:'SEMAR', 12:'SSA', 13:'UNIVERSITARY', 99:float('nan')} 

df['SECTOR'] = df['SECTOR'].map(Dic_Sector)
print(df.SECTOR.value_counts()) # A brief information
# Reformatting the variables related to Entity area.
Dic_Entity = {1:'AGUASCALIENTES',2:'BAJA CALIFORNIA',3:'BAJA CALIFORNIA SUR',4:'CAMPECHE',5:'COAHUILA DE ZARAGOZA',
              6:'COLIMA',7:'CHIAPAS',8:'CHIHUAHUA',9:'MEXICO CITY',10:'DURANGO',11:'GUANAJUATO',12:'GUERRERO',
              13:'HIDALGO',14:'JALISCO',15:'STATE OF MEXICO',16:'MICHOACÁN DE OCAMPO',17:'MORELOS',18:'NAYARIT',19:'NUEVO LEÓN',
              20:'OAXACA',21:'PUEBLA',22:'QUERÉTARO',23:'QUINTANA ROO',24:'SAN LUIS POTOSÍ',25:'SINALOA',26:'SONORA',
              27:'TABASCO',28:'TAMAULIPAS',29:'TLAXCALA',30:'VERACRUZ',31:'YUCATÁN',32:'ZACATECAS',
              36:'UNITED MEXICAN STATES',97:'NOT APLIED',98:'IGNORED',99:float('nan')}

df['ENTITY_UM'] = df['ENTITY_UM'].map(Dic_Entity)
df['ENTITY_RES'] = df['ENTITY_RES'].map(Dic_Entity)
df['ENTITY_NAT'] = df['ENTITY_NAT'].map(Dic_Entity)
print(df.ENTITY_UM.value_counts()) # A brief information
Dic_Sex = {1:'Female', 2:'Male', 99:float('nan')}
df['SEX'] = df['SEX'].map(Dic_Sex)
print(df.SEX.value_counts()) # A brief information
Dic_Patient_Type = {1:'NOT HOSPITALIZED', 2:'HOSPITALIZED', 99:float('nan')}
df['PATIENT_TYPE'] = df['PATIENT_TYPE'].map(Dic_Patient_Type)
print(df.PATIENT_TYPE.value_counts()) # A brief information
### Reformate the variables which were coded in binary form
# Dic_Yes_No = {1:'Yes', 2:'No',  97:'NOT APLIED',98:'IGNORED',99:float('nan')}
Dic_Yes_No = {1:'Yes', 2:'No',  97:'NOT APLIED',98:float('nan'),99:float('nan')} # Ignored is considered as missing data
Dic_Pregnant = {1:'Yes', 2:'No',  97:float('nan'),98:float('nan'),99:float('nan')}
df['INTUBATED'] = df['INTUBATED'].map(Dic_Yes_No) 
df['PNEUMONIA'] = df['PNEUMONIA'].map(Dic_Yes_No)
df['PREGNANT'] = df['PREGNANT'].map(Dic_Pregnant)
df['SPEAK_INDIGENOUS_LANGUAGE'] = df['SPEAK_INDIGENOUS_LANGUAGE'].map(Dic_Yes_No)
df['DIABETES'] = df['DIABETES'].map(Dic_Yes_No)
df['COPD'] = df['COPD'].map(Dic_Yes_No)
df['ASTHMA'] = df['ASTHMA'].map(Dic_Yes_No)
df['INMUSUPR'] = df['INMUSUPR'].map(Dic_Yes_No)
df['HYPERTENSION'] = df['HYPERTENSION'].map(Dic_Yes_No)
df['OTHER_DISEASE'] = df['OTHER_DISEASE'].map(Dic_Yes_No)
df['CARDIOVASCULAR'] = df['CARDIOVASCULAR'].map(Dic_Yes_No)
df['OBESITY'] = df['OBESITY'].map(Dic_Yes_No)
df['CHRONIC_KIDNEY_DISEASE'] = df['CHRONIC_KIDNEY_DISEASE'].map(Dic_Yes_No)
df['SMOKE'] = df['SMOKE'].map(Dic_Yes_No)
df['OTHER_CASE_CONTACT'] = df['OTHER_CASE_CONTACT'].map(Dic_Yes_No)
df['MIGRANT'] = df['MIGRANT'].map(Dic_Yes_No)
df['UCI'] = df['UCI'].map(Dic_Yes_No)
df['INDIGENOUS'] = df['INDIGENOUS'].map(Dic_Yes_No)
df['TESTED'] = df['TESTED'].map(Dic_Yes_No)
Dic_Nationality = {1:'Mexican', 2:'Foreigner', 99:float('nan')}
df['NATIONALITY'] = df['NATIONALITY'].map(Dic_Nationality)
print(df.NATIONALITY.value_counts()) # A brief information
Dic_Result = {1:'Positive SARS-CoV-2', 2:'Non-Positive SARS-CoV-2', 3:'pending result', 4:'Inadequate Result',5:'Not applied'}
df['RESULT_LAB'] = df['RESULT_LAB'].map(Dic_Result)
print(df.RESULT_LAB.value_counts()) # A brief information
Dic_Clasif = {1:'Confirmed by epidemiological clinical association', 2:'Confirmed by ruling committee',
             3:'Confirmed by laboratory test', 4:'Invalid by laboratory', 5:'Not tested by laboratory',
             6:'Suspected case', 7:'Negative tested by laboratory'}
df['FINAL_CLASIFICATION'] = df['FINAL_CLASIFICATION'].map(Dic_Clasif)
df['FINAL_CLASIFICATION'].value_counts()
df['DEATH_DATE'] = pd.to_datetime(df['DEATH_DATE'])<jupyter_output><empty_output><jupyter_text>**You could preserve the original file by following codes:**<jupyter_code>df.to_csv('CovMexico_lastestdata.csv', sep=',', encoding='utf-8')<jupyter_output><empty_output><jupyter_text>Data preparation for "Clustering Analysis"  In this section, we establish some extra columns by reformatting existent variables which are essential in order for clustering analysis.   Preprocessing of comorbidities and outcomes<jupyter_code>import pandas as pd
import numpy as np
df = pd.read_csv("CovMexico_lastestdata.csv", delimiter = ',') # Read the datasheet which we formatted before
df = df[(df['RESULT_LAB']=='Positive SARS-CoV-2')|(df['RESULT_LAB']=='Non-Positive SARS-CoV-2')] # Only positive and negative patients

def Outcome(date):
    # As there is only date of Death, we cannot deduce whether the patient is recovered or still in the hospital
    if pd.isna(date): # If there is not decease date about the patient, we consider them as non-deceased individual
        return "Non-Deceased" 
    return "Deceased"

# df = df[df["RESULT"] == "Positive SAR-CoV-2"]  # Obtanin only the patients who were tested as positive with SAR-CoV-2
df["OUTCOME"] = df["DEATH_DATE"].map(Outcome)
print(df.OUTCOME.value_counts())<jupyter_output>Non-Deceased    2283433
Deceased         131449
Name: OUTCOME, dtype: int64<jupyter_text>Convert the required variables, for clustering analysis, into **Binary** format.<jupyter_code>Dic_Yes_No = {'Yes':str(1), 'No':str(0),'NOT APLIED':str(0)}
Dic_Yes_No2 = {'Yes':str(1), 'No':str(0)}

# We reformate these variables into Binary formz in order to use clustering.
df['INTUBATED'] = df['INTUBATED'].map(Dic_Yes_No) 
df['PNEUMONIA'] = df['PNEUMONIA'].map(Dic_Yes_No)

df['SPEAK_INDIGENOUS_LANGUAGE'] = df['SPEAK_INDIGENOUS_LANGUAGE'].map(Dic_Yes_No)
df['DIABETES'] = df['DIABETES'].map(Dic_Yes_No)
df['COPD'] = df['COPD'].map(Dic_Yes_No)
df['ASTHMA'] = df['ASTHMA'].map(Dic_Yes_No)
df['INMUSUPR'] = df['INMUSUPR'].map(Dic_Yes_No)
df['HYPERTENSION'] = df['HYPERTENSION'].map(Dic_Yes_No)
df['OTHER_DISEASE'] = df['OTHER_DISEASE'].map(Dic_Yes_No)
df['CARDIOVASCULAR'] = df['CARDIOVASCULAR'].map(Dic_Yes_No)
df['OBESITY'] = df['OBESITY'].map(Dic_Yes_No)
df['CHRONIC_KIDNEY_DISEASE'] = df['CHRONIC_KIDNEY_DISEASE'].map(Dic_Yes_No)
df['SMOKE'] = df['SMOKE'].map(Dic_Yes_No)
df['UCI'] = df['UCI'].map(Dic_Yes_No)

df['OTHER_CASE_CONTACT'] = df['OTHER_CASE_CONTACT'].map(Dic_Yes_No2)
df['OTHER_CASE_CONTACT'].fillna('0', inplace=True)
df['INDIGENOUS'] = df['INDIGENOUS'].map(Dic_Yes_No)
df['TESTED'] = df['TESTED'].map(Dic_Yes_No)
df['PREGNANT'] = df['PREGNANT'].map(Dic_Yes_No2)
df['INDIGENOUS'] = df['INDIGENOUS'].map(Dic_Yes_No)


df['PREGNANT'].fillna('0', inplace=True)
df['UCI'].fillna('0', inplace=True)
### Obtain the patients of the first wave in Mexico who presented at least one month the symptoms
from datetime import date, timedelta
import time

df["SYMPTOMS_DATE"] = pd.to_datetime(df["SYMPTOMS_DATE"])
df = df[(df['SYMPTOMS_DATE'] < '2020-10-1')]<jupyter_output><empty_output><jupyter_text>Remove some missing data related to the following variables to prevent analysis from disturbance.<jupyter_code>df = df.dropna(subset = ['INTUBATED','PNEUMONIA','DIABETES','COPD', 'ASTHMA','INMUSUPR',
       'HYPERTENSION','OTHER_DISEASE','CARDIOVASCULAR','OBESITY','CHRONIC_KIDNEY_DISEASE','SMOKE'])<jupyter_output><empty_output><jupyter_text>The new variable **Survival days** is calculated by the **date of death** minus the **date of Symptoms**<jupyter_code>df["DEATH_DATE"] = pd.to_datetime(df["DEATH_DATE"])
df["Survival_days"] = df["DEATH_DATE"] - df["SYMPTOMS_DATE"]

# Convert date into days number
df["Survival_days"] = df.Survival_days.apply(lambda x: x.days)
df[df['Survival_days']<0]
df = df[(df['Survival_days']>0)|(df['Survival_days'].isnull())]
len(df)<jupyter_output><empty_output><jupyter_text>The new variable **FromSymptomToHospital days** is calculated by the **date of Admission**  minus the **date of Symptoms** (only for the patients who were hospitalized)<jupyter_code>df['ADMISSION_DATE'] = pd.to_datetime(df['ADMISSION_DATE'])
df["FromSymptomToHospital_days"] = df['ADMISSION_DATE'] - df["SYMPTOMS_DATE"]

# Convert date into days number
df["FromSymptomToHospital_days"] = df[df['PATIENT_TYPE'] == 'HOSPITALIZED'].FromSymptomToHospital_days.apply(lambda x: x.days)
del df['Unnamed: 0'] # Remove duplicated index columns
df1 = df[df['RESULT_LAB']=='Positive SARS-CoV-2']<jupyter_output><empty_output><jupyter_text>Save the dataset into csv file<jupyter_code>df.to_csv('COVID19_MEXICO_LATEST_DATA_POPULATION.csv', sep=',', encoding='utf-8') # Both valid positive and Negative patients.
df1.to_csv('COVID19_MEXICO_LATEST_DATA.csv', sep=',', encoding='utf-8')           # Only valid positive patients<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14489247311827957, 'fraction_numerical': 0.017114695340501793, 'mean_word_length': 3.904636343660734, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 52, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '596423', 'license': 'Apache-2.0', 'n_tokens_mistral': 8134, 'n_tokens_neox': 7451, 'n_words': 1684, 'path': 'CovMexico_Data_Preprocessing.ipynb', 'repo_name': 'bdslab-upv/covid19-metaclustering', 'size': '22320'}