<jupyter_start><jupyter_text>MARKOVID> Modeling and Simulation                                     *February 2022*---Project partners:*   <NAME>*   <NAME>*   <NAME>Professor: <NAME>![](https://raw.githubusercontent.com/Chalysis/Markovid/main/cadena_probs.png)<jupyter_code># Download dataset
! wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1-KDj7eWXJ63X4lqgm1BJdFEIbF19xdrw' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1-KDj7eWXJ63X4lqgm1BJdFEIbF19xdrw" -O theDataset.csv && rm -rf /tmp/cookies.txt

#Required libraries
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import collections
import math<jupyter_output><empty_output><jupyter_text>The first step is to load the  data located in the csv downloaded earlier.<jupyter_code>cols = ['FECHA_ACTUALIZACION', 'ID_REGISTRO',  'TIPO_PACIENTE', 'FECHA_INGRESO', 
        'FECHA_SINTOMAS', 'FECHA_DEF', 'CLASIFICACION_FINAL']
covidato = pd.read_csv('/content/theDataset.csv', usecols=cols, compression='zip'); covidato
#We proceed to review the structure.
covidato.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 7627400 entries, 0 to 7627399
Data columns (total 7 columns):
 #   Column               Dtype 
---  ------               ----- 
 0   FECHA_ACTUALIZACION  object
 1   ID_REGISTRO          object
 2   TIPO_PACIENTE        int64 
 3   FECHA_INGRESO        object
 4   FECHA_SINTOMAS       object
 5   FECHA_DEF            object
 6   CLASIFICACION_FINAL  int64 
dtypes: int64(2), object(5)
memory usage: 407.3+ MB<jupyter_text>Model StructureTo make the chain, we first need to define the states. After a review, the states are as follows:* Symptomatic (S)      (Sintomático)* Negative (N)         (Negativo)* Positive (P)         (Positivo)* Hospitalized (H)     (Hospitalizado)* Deceased (D)         (Difunto)In order to be able to see the behavior of these states correctly, it is necessary to make the separations.<jupyter_code>#Sets
neg = covidato[covidato.CLASIFICACION_FINAL == 7] #Negativa
pos = covidato[covidato.CLASIFICACION_FINAL.isin([1,2,3])] # Positiva
hos = covidato[(covidato.TIPO_PACIENTE == 2)] #Hospitalizada
dif = covidato[covidato.FECHA_DEF != "9999-99-99"] #Difunta<jupyter_output><empty_output><jupyter_text>What follows is to consider the intersections between two sets.<jupyter_code>### ######## ###
### NEGATIVE ###
### ######## ###
# People who have had one negative and one positive result
pos_neg = neg[neg.ID_REGISTRO.isin(pos.ID_REGISTRO)]
# People who have been hospitalized and do not have COVID
hos_neg = hos[hos.ID_REGISTRO.isin(neg.ID_REGISTRO)]
### ######## ###
### POSITIVE ###
### ######## ###
# People who have been hospitalized and have COVID
hos_pos = hos[hos.ID_REGISTRO.isin(pos.ID_REGISTRO)]
#Deceased persons who suffered from covid
dif_pos = dif[dif.ID_REGISTRO.isin(pos.ID_REGISTRO)]
dif[dif.ID_REGISTRO.isin(pos_neg.ID_REGISTRO)].sort_values(by='ID_REGISTRO')
dif[dif.ID_REGISTRO=='adba70'][['ID_REGISTRO','FECHA_ACTUALIZACION','FECHA_INGRESO','FECHA_SINTOMAS','FECHA_DEF','CLASIFICACION_FINAL']]
dif.FECHA_DEF.max() 
rec = pos[(pos.FECHA_INGRESO < '2021-11-13')&(~pos.ID_REGISTRO.isin(dif.ID_REGISTRO))]['ID_REGISTRO']
inf_gnr = {'pT': covidato.ID_REGISTRO.nunique(), 'casos': pos.ID_REGISTRO.nunique(), 
           'muertes': dif_pos.ID_REGISTRO.nunique(), 'hospitalizados': hos_pos.ID_REGISTRO.nunique(),
           'No hospitalizados': pos.ID_REGISTRO.nunique() - hos_pos.ID_REGISTRO.nunique(), 
           'recuperados': rec.nunique()
}
inf_gnr
dif.TIPO_PACIENTE.value_counts()<jupyter_output><empty_output><jupyter_text>Probabilities<jupyter_code>######################
### PROBABILITIES ###
######################

### Being in state S###

#S->N
P_S2N = neg.ID_REGISTRO.nunique()/inf_gnr['pT'] #P(N&S)/P(S)


#S->P
P_S2P = pos.ID_REGISTRO.nunique()/inf_gnr['pT'] #P(P&S)/P(S)

print(f"S->N={P_S2N:.4f}, S->P{P_S2P:.4f}")
### Being in state P ###

#P-> N
P_P2N = pos[(pos.ID_REGISTRO.isin(rec))&(~pos.ID_REGISTRO.isin(hos.ID_REGISTRO))].ID_REGISTRO.nunique()/pos.ID_REGISTRO.nunique()

#P -> H
P_P2H = hos_pos.ID_REGISTRO.nunique()/pos.ID_REGISTRO.nunique(); P_P2H

#P -> D
P_P2M = pos[(pos.ID_REGISTRO.isin(dif.ID_REGISTRO))&(~pos.ID_REGISTRO.isin(hos.ID_REGISTRO))].ID_REGISTRO.nunique()/pos.ID_REGISTRO.nunique()

print(f"P->N={P_P2N:.4f},P->H={P_P2H:.4f},P->M={P_P2M:.4f}")
### Being in state H ###

P_H2N = hos_pos[(hos_pos.ID_REGISTRO.isin(rec))].ID_REGISTRO.nunique()/hos_pos.ID_REGISTRO.nunique()

P_H2M = hos_pos[(hos_pos.ID_REGISTRO.isin(dif.ID_REGISTRO))].ID_REGISTRO.nunique()/hos_pos.ID_REGISTRO.nunique()

print(f"H->N={P_H2N:.4f},H->M={P_H2M:.4f}")
#Dictionary definition

chain = {'S':[('P',0.2676), ('N',0.7061)], 
           'P':[('N',0.8402), ('H',0.1548), ('D',0.0050)], 
           'H':[('N',0.5101), ('D',0.4899)],
           'D':[('D',1)]} #Final state<jupyter_output><empty_output><jupyter_text>We begin the simulations We create a function that allows us to obtain the next state of a certain population<jupyter_code>def get_states(people):
    #people: List of strings that represents thecurrent state of a population
    """
    This function will see the current state for each person and will update it,
    at the end the function will count the people in each state and will return
    a dictionary with the counter
    """

    n = len(people)

    for i in range(n):
        state = people[i]
        
        #If the person doesn't have covid, we can't predict their next state
        if state =='N':
            continue

        #We generate a random number in the interval [0,1)
        x = np.random.random()

        #We get the transitions and sort them ascending order(?) by the probaility
        transitions = chain[state]
        transitions.sort(key = lambda x: x[1])

        #We get the probabilities
        probs = [t[1] for t in transitions]

        #We get the next state of our person by checking the interval where the
        #random number fell (?) 
        for j in range(len(probs)):
            if j==0:
                if x <= probs[j]:
                    people[i] = transitions[j][0]
                    break
                continue

            lim_inf = probs[j-1]
            lim_sup = probs[j]+lim_inf
                
            if x > lim_inf and x <= lim_sup:
                people[i] = transitions[j][0]
                break

    #We return the dictionary that counts the results
    counter = dict(collections.Counter(people))
    #Our mistake will be the people that didn't get a diagnosis
    try:
        counter['e']= counter['S']
    except:
        counter['e']=0
    #Our S value will be the quantity of people that did a test
    counter['S']=n

    return counter
#Test
pe = ['S' for i in range(1000)]
get_states(pe)<jupyter_output><empty_output><jupyter_text>We then create a function to handle the simulations<jupyter_code>def do_simulations(s_people,initial_people=[]):
    #s_people: list with integers of people per day
    #initial_people: list with strings that represent the current state of all people
    """
    This function does n simulations with p people added in each one, the result
    is a dataframe with the main details
    """

    
    if initial_people==[]:
        prev_people=['S']*s_people[0]

    # We create a dataframe that will help us to keep a track
    simulations = pd.DataFrame(columns=['S','N','P','H','D','e'])

    #The first simulation
    simulations.loc[1] = get_states(prev_people)
    simulations.fillna(0,inplace=True)
    #Now we start the simulations
    for i,p in enumerate(s_people[1:]):
        
        #We create our people to check
        positive = int(simulations.loc[i+1,'P'])
        hospitalized = int(simulations.loc[i+1,'H'])
        error = int(simulations.loc[i+1,'e'])
        people = ['S']*(p+error) + ['P']*positive + ['H']*hospitalized
        
        #We store the results
        simulations.loc[i+2] = get_states(people)
        simulations.fillna(0,inplace=True)

    #We return the dataframe
    return simulations
s_people = list(covidato['FECHA_SINTOMAS'].value_counts())
test1 = do_simulations(s_people)
#test1=np.log2(test1)
test1
sims=list(range(1,test1.shape[0]+1))
sns.lineplot(data=test1, x=sims, y="N", color='blue', label="Negative")
sns.lineplot(data=test1, x=sims, y="P", color='green', label="Positive")
sns.lineplot(data=test1, x=sims, y="H", color='goldenrod', label="Hospitalised")
sns.lineplot(data=test1, x=sims, y="D", color='red', label="Death rate")
plt.title("Simulation results")
plt.xlabel('Simulation')
plt.ylabel('People')
plt.legend()
plt.show()
sims=list(range(1,test1.shape[0]+1))
sns.lineplot(x=sims, y=test1.P.cumsum(), color='green', label="Positive")
sns.lineplot(x=sims, y=test1.H.cumsum(), color='goldenrod', label="Hospitalised")
sns.lineplot(x=sims, y=test1.D.cumsum(), color='red', label="Death rate")

plt.xlabel('Simulation')
plt.ylabel('People')
plt.show()
sims=list(range(1,test1.shape[0]+1))
sns.lineplot(data=test1, x=sims, y="N", color='blue')
plt.title('Negative')
plt.xlabel('Simulation')
plt.ylabel('People')
plt.show()
sims=list(range(1,test1.shape[0]+1))
sns.lineplot(data=test1, x=sims, y="P", color='green')
plt.title('Positive')
plt.xlabel('Simulation')
plt.ylabel('People')
plt.show()
sims=list(range(1,test1.shape[0]+1))
sns.lineplot(data=test1, x=sims, y="H", color='goldenrod')
plt.title('Hospitalised')
plt.xlabel('Simulation')
plt.ylabel('People')
plt.show()
sims=list(range(1,test1.shape[0]+1))
D=list(dif[dif.FECHA_ACTUALIZACION=='2022-01-19'].FECHA_DEF.value_counts())
nd = [i for i in range(1,len(D)+1)]
sns.lineplot(x=nd, y=D, color='darkred', label='Real')
sns.lineplot(data=test1, x=sims, y="D", color='red', label = 'Simulation')
plt.legend()
plt.title("Results for dead people")
plt.xlabel('Simulation')
plt.ylabel('People')
plt.show()
sims=list(range(1,test1.shape[0]+1))
sns.lineplot(data=test1, x=sims, y="e", color='darkorchid')
plt.title('Error')
plt.xlabel('Simulation')
plt.ylabel('People')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16258417097687128, 'fraction_numerical': 0.021079340294720406, 'mean_word_length': 4.180990899898887, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 47, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '251577', 'license': 'CC0-1.0', 'n_tokens_mistral': 4032, 'n_tokens_neox': 3818, 'n_words': 772, 'path': 'MarKovid.ipynb', 'repo_name': 'ProbModel/Markovid', 'size': '10301'}