<jupyter_start><jupyter_text>Read csv files<jupyter_code>covid19 = pd.read_csv("../../../data/output/covid.csv")
hospital= pd.read_csv("../../../data/output/COVID19_HOSBEDS_MERGE.CSV")
employment = pd.read_csv("../../../data/output/COVID19_EMPLOYMENT_MERGE.CSV")<jupyter_output><empty_output><jupyter_text>Michigan State is chosen for the analysis<jupyter_code>covid19 = covid19.drop(covid19.loc[covid19['County Name'] == "Statewide Unallocated"].index)
MI_counties =covid19.loc[covid19['State'] == 'MI']
MI_state = MI_counties.groupby(['State']).sum()
MI_counties_cases = covid19.loc[:, covid19.columns.str.endswith("_x")]<jupyter_output><empty_output><jupyter_text>These next sections get the normalized daily new cases and deaths in Michigan State<jupyter_code>MI_state_cases= MI_state.loc[:, MI_state.columns.str.endswith("_x")]
#select from the first day of infection 
MI_state_cases =MI_state_cases.loc[: ,'3/9/20_x':]
MI_new_cases= MI_state_cases.diff(axis=1)
MI_new_cases= MI_new_cases.drop(columns= ['3/9/20_x'], axis=1)
MI_new_cases.columns = MI_new_cases.columns.str.replace('_x', '')
MI_new_cases

normalized_daily_new_cases_MI= round(((MI_new_cases.div(9986857)).multiply(1_000_000)))
daily_mean_cases= normalized_daily_new_cases_MI.mean(axis=1)
print(daily_mean_cases)

MI_state_deaths = MI_state.loc[:, MI_state.columns.str.endswith("_y")]
#select from the first day of infection 
MI_state_deaths =MI_state_deaths.loc[: ,'3/9/20_y':]
MI_new_deaths= MI_state_deaths.diff(axis=1)
MI_new_deaths= MI_new_deaths.drop(columns= ['3/9/20_y'], axis=1)
MI_new_deaths.columns = MI_new_deaths.columns.str.replace('_y', '')
MI_new_deaths


normalized_daily_new_deaths_MI= round(((MI_new_deaths.div(9986857)).multiply(20_000_000)))
daily_mean_deaths = normalized_daily_new_deaths_MI.mean(axis=1)
print(daily_mean_deaths)
normalized_daily_new_deaths_MI<jupyter_output>State
MI    65.138211
dtype: float64<jupyter_text>Fitting a linear model for MI state: store the days in X-axis and the number of cases/ deaths in Y-axis. To make the analysis easier, the days have been taking as integers(day1, day2...)<jupyter_code>X_cases = MI_new_cases.columns.tolist()
X_cases = pd.to_datetime(X_cases) 
Y_cases = normalized_daily_new_cases_MI.iloc[0].values.tolist()
X = [i for i in range(MI_new_cases.columns.size)]

# next  seven days for prediction  
K = [i for i in range(MI_new_cases.columns.size ,MI_new_cases.columns.size + 7)]<jupyter_output><empty_output><jupyter_text>Convert the data into 2D arrays as the linear model take 2Dimension data<jupyter_code>Xcases_train = np.array(X)
Ycases_train = np.array(Y_cases)
K_test = np.array(K)

Xcases_train = Xcases_train[:, np.newaxis]
Ycases_train = Ycases_train[:, np.newaxis]
K_test = K_test[:, np.newaxis]<jupyter_output><empty_output><jupyter_text>In this section,  a linear regression has been modelled. A prediction on the same data that was used to trained was made to see the performance of the model. Another prediction mas made for the next seven days.<jupyter_code>model = LinearRegression()
model.fit(Xcases_train, Ycases_train) #fit model 

y_hat1 = model.predict(Xcases_train)
y_hat = model.predict(K_test)
print("The mean of predicted cases: ", y_hat.mean())
print("The RMSE error :", np.sqrt(metrics.mean_squared_error(Ycases_train, y_hat1)))
print ("The R^2: ",r2_score(Ycases_train, y_hat1))
plt.figure()
plt.plot(Xcases_train.squeeze(), Ycases_train, 'o')
plt.plot(Xcases_train.squeeze(), y_hat1)
plt.plot(K_test,y_hat);
plt.xlabel('Number of days')
plt.ylabel('New cases MI')
plt.title('Linear Regression MI-cases')<jupyter_output><empty_output><jupyter_text>The mean of daily new cases in Michigan was 96.23, that is to say there is an average of about 96 new cases each day and our model have predicted a mean of 190.  The performance of the linear model on this data is not very good. Probabily because a linear model does not best fit the data . Moreover we can see that through the R^2 measure which was only 0.14.   Prediction for number of deaths in Michigan State<jupyter_code>Y_deaths = normalized_daily_new_deaths_MI.iloc[0].values.tolist()
X_deaths_train = np.array(X)
Y_deaths_train = np.array(Y_deaths)
Kdeaths_test = np.array(K)

X_deaths_train = X_deaths_train[:, np.newaxis]
Y_deaths_train = Y_deaths_train[:, np.newaxis]
Kdeaths_test = Kdeaths_test[:, np.newaxis]
model = LinearRegression()
model.fit(X_deaths_train, Y_deaths_train) #fit model 

y_pred = model.predict(Kdeaths_test)
y_pred1 = model.predict(X_deaths_train)
print("The mean of predicted deaths: ", y_pred.mean())
y_pred
print("The RMSE error :", np.sqrt(metrics.mean_squared_error(Y_deaths_train, y_pred1)))
print ("The R^2: ",r2_score(Y_deaths_train, y_pred1))
plt.figure()
plt.plot(X_deaths_train.squeeze(), Y_deaths_train, 'o')
plt.plot(X_deaths_train.squeeze(), y_pred1)
plt.plot(Kdeaths_test.squeeze(), y_pred);
plt.xlabel('Number of days')
plt.ylabel('New deaths MI')
plt.title('Linear Regression MI-deaths')<jupyter_output><empty_output><jupyter_text>For the number of deaths, similar to number of cases, the linear model is not very accurate we would have expect around 30-50 deaths daily based on the average, it would have been acceptable. In addition, the R^2 metric was only 0.13, so the model is not a good model for this data.  Non-Linear Regression Model   Polynomial Model: Since they linear regression model did not work let's a non linear in this case a polynomial model and observe its performance.<jupyter_code>X_train= np.array(X)
Y_train= np.array(Y_cases)
X_train = X_train[:, np.newaxis]
Y_train= Y_train[:, np.newaxis]

poly = PolynomialFeatures(degree = 5) 
X_poly = poly.fit_transform(X_train) 

poly.fit(X_poly, Y_train) 
lin2 = LinearRegression() 
lin2.fit(X_poly, Y_train) 

ycases_hat = lin2.predict(poly.fit_transform(X_train))

plt.scatter(X_train, Y_train) 
plt.plot(X_train, ycases_hat, color = 'orange')

#Trend line 
X = [i for i in range(MI_new_cases.columns.size)]
Y_cases = normalized_daily_new_cases_MI.iloc[0].values.tolist()
X = np.array(X)
Y = np.array(Y_cases)
z = np.polyfit(X, Y, 1)
p = np.poly1d(z)

plt.plot(X,p(X),"r-")

plt.xlabel('Number of days')
plt.ylabel('New cases MI')
plt.title('Polynomial Regression MI-cases')
print ("The RMSE error: ", np.sqrt(metrics.mean_squared_error(Ycases_train, lin2.predict(poly.fit_transform(X_train)))))
print ("The R^2: ",r2_score(Ycases_train, lin2.predict(poly.fit_transform(X_train))))
Y_deaths = normalized_daily_new_deaths_MI.iloc[0].values.tolist()
X_deaths_train = np.array(X)
Y_deaths_train = np.array(Y_deaths)
Kdeaths_test = np.array(K)

X_deaths_train = X_deaths_train[:, np.newaxis]
Y_deaths_train= Y_deaths_train[:, np.newaxis]
Kdeaths_test= Kdeaths_test[:, np.newaxis]


poly = PolynomialFeatures(degree = 7) 
X_poly = poly.fit_transform(X_train) 

poly.fit(X_poly, Y_deaths_train) 
lin2 = LinearRegression() 
lin2.fit(X_poly, Y_deaths_train) 

ycases_hat = lin2.predict(poly.fit_transform(Kdeaths_test))
ycases_hat1 = lin2.predict(poly.fit_transform(X_train))

plt.scatter(X_train, Y_deaths_train) 
plt.plot(X_train, ycases_hat1, color = 'orange')

X = [i for i in range(MI_new_cases.columns.size)]
Y_cases = normalized_daily_new_deaths_MI.iloc[0].values.tolist()
X = np.array(X)
Y = np.array(Y_cases)
z = np.polyfit(X, Y, 1)
p = np.poly1d(z)

plt.plot(X,p(X),"r-")

plt.xlabel('Number of days')
plt.ylabel('New deaths MI')
plt.title('Polynomial Regression MI-deaths')
print ("The RMSE error: ", np.sqrt(metrics.mean_squared_error(Y_deaths_train, lin2.predict(poly.fit_transform(X_train)))))
print ("The R^2: ",r2_score(Y_deaths_train, lin2.predict(poly.fit_transform(X_train))))<jupyter_output>The RMSE error:  62.66449435856421
The R^2:  0.5401060326673048<jupyter_text>The top 5 counties from stage 2 were: Wayne county, Oakland County,  Macomb County,  Genesee County and Kent County<jupyter_code>MI_counties_cases= MI_counties.loc[:, MI_counties.columns.str.endswith("_x")]

#select from the first day of infection 
MI_counties_cases = MI_counties_cases.loc[: ,'3/9/20_x':]
MI_counties_new_cases= MI_counties_cases.diff(axis=1)
MI_counties_new_cases= MI_counties_new_cases.drop(columns= ['3/9/20_x'], axis=1)
MI_counties_new_cases.columns = MI_counties_new_cases.columns.str.replace('_x', '')
MI_counties_new_cases.head()
MI_counties_new_cases.columns = pd.to_datetime(MI_counties_new_cases.columns.tolist())
MI_counties_new_cases.head()
# convert date to int to facilitate fitting the model 
X_counties = [i for i in range(MI_counties_new_cases.columns.size)]
top5_counties = MI_counties_new_cases.loc[[1336, 1317, 1304,1295,1279]] 
top5_counties
for i in range (top5_counties.index.size):
    Y = top5_counties.iloc[i].values.tolist()
    X_train= np.array(X)
    Y_train= np.array(Y)
    X_train = X_train[:, np.newaxis]
    Y_train= Y_train[:, np.newaxis]

    model = LinearRegression()
    model.fit(X_train,Y_train)

    y_hat= model.predict(X_train)
    y_hat1= model.predict(K_test)
    
    #plt.figure()
    #plt.plot(X_train.squeeze(), Y_train, 'o')
    #plt.plot(X_train.squeeze(), y_hat);
    #plt.plot(K_test.squeeze(), y_hat1)<jupyter_output><empty_output><jupyter_text>For the 5 top counties, a polynimal model fits the data better.<jupyter_code>for i in range (top5_counties.index.size):
    Y = top5_counties.iloc[i].values.tolist()
    X_train= np.array(X)
    Y_train= np.array(Y)
    X_train = X_train[:, np.newaxis]
    Y_train= Y_train[:, np.newaxis]
    
    poly = PolynomialFeatures(degree = 5) 
    X_poly = poly.fit_transform(X_train) 
    poly.fit(X_poly, Y_train) 
    lin2 = LinearRegression() 
    lin2.fit(X_poly, Y_train) 
    y_hat = lin2.predict(poly.fit_transform(X_train))
    y_hat2 = lin2.predict(poly.fit_transform(K_test))
    
    Y_cases = top5_counties.iloc[i].values.tolist()
    Y = np.array(Y_cases)
    z = np.polyfit(X, Y, 1)
    p = np.poly1d(z)
    
    plt.figure()
    plt.plot(X_train.squeeze(), Y_train, 'o')
    plt.plot(X_train.squeeze(), y_hat);
    plt.plot(K_test.squeeze(), y_hat2)
    plt.plot(X,p(X),"r-")<jupyter_output><empty_output><jupyter_text>On all the graphs we can see the trend line in red along with the prediction path in orange. the green part of the curve represents the prediction for the next seven days.   Point of No Return<jupyter_code>#get the ICU beds column 
MI_counties_hos =hospital.loc[hospital['State'] == 'MI']
MI_state_hos = MI_counties_hos.groupby(['State']).sum()
total_ICU_bed = MI_state_hos['NUM_ICU_BEDS']
total_ICU_bed<jupyter_output><empty_output><jupyter_text>Through the non-linear model, the trend in daily new death was small with a relative number of about maximum 5 deaths per day. This value not close to the total number of  ICU bed in Michigan State which indicates that the state is doing well and there is no "point of no return".   Decision Tree<jupyter_code>from sklearn import tree
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor

Xcases_trainD = np.array(X)
Ycases_trainD = np.array(Y_cases)
K_test = np.array(K)
Xcases_trainD = Xcases_trainD[:, np.newaxis]
Ycases_trainD = Ycases_trainD[:, np.newaxis]
K_test = K_test[:, np.newaxis]
clf = DecisionTreeRegressor()
clf = clf.fit(X_train, Y_train)

yhat_test = clf.predict(X_train)
yhat_test1 = clf.predict(K_test)

print("The RMSE: ",np.sqrt(metrics.mean_squared_error(Y_train, yhat_test)))
score = r2_score(Y_train, yhat_test)
print("R^2:",score)
plt.figure()
plt.plot(X_train.squeeze(), Y_train, 'o')
plt.plot(X_train.squeeze(), yhat_test);

Y_cases = normalized_daily_new_cases_MI.iloc[0].values.tolist()
X = np.array(X)
Y = np.array(Y_cases)
z = np.polyfit(X, Y, 1)
p = np.poly1d(z)

plt.plot(X,p(X),"r-")
plt.xlabel('Number of days')
plt.ylabel('New cases MI')
plt.title('Decision Tree MI-cases')<jupyter_output><empty_output><jupyter_text>Decision Tree for deaths<jupyter_code>Y_deaths = normalized_daily_new_deaths_MI.iloc[0].values.tolist()
X_deaths_train = np.array(X)
Y_deaths_train = np.array(Y_deaths)
Kdeaths_test = np.array(K)

X_deaths_train = X_deaths_train[:, np.newaxis]
Y_deaths_train = Y_deaths_train[:, np.newaxis]
Kdeaths_test = Kdeaths_test[:, np.newaxis]

clf = DecisionTreeRegressor()
clf = clf.fit(X_deaths_train, Y_deaths_train)

yhat_test = clf.predict(X_deaths_train)
yhat_test1 = clf.predict(K_test)

print("The RMSE: ",np.sqrt(metrics.mean_squared_error(Y_deaths_train, yhat_test)))
score = r2_score(Y_deaths_train, yhat_test)
print("R^2:",score)

plt.figure()
plt.plot(X_train.squeeze(), Y_train, 'o')
plt.plot(X_train.squeeze(), yhat_test);

Y_cases = normalized_daily_new_deaths_MI.iloc[0].values.tolist()
X = np.array(X)
Y = np.array(Y_deaths)
z = np.polyfit(X, Y, 1)
p = np.poly1d(z)

plt.plot(X,p(X),"r-")

plt.xlabel('Number of days')
plt.ylabel('New deaths MI')
plt.title('Decision Tree MI-deaths')<jupyter_output><empty_output><jupyter_text>The R^2 for both the number of cases and the number of deaths is 1 which indicates that the model performes well and the RMSE is small compare to the linear and polynomial model. From this analysis we can conclude that Decision Tree fits best our data.   Random Forest<jupyter_code>X_train= np.array(X)
Y_train= np.array(Y_cases)
X_train = X_train[:, np.newaxis]
Y_train= Y_train[:, np.newaxis]

model1 = RandomForestRegressor()
model1.fit(X_train, Y_train)

yhat_RF = model1.predict(X_train)
yhat_RF1 = model1.predict(K_test)

print("The RMSE:",np.sqrt(metrics.mean_squared_error(Y_train, yhat_RF)))
score1 = r2_score(Y_train, yhat_RF)
print("R^2: ", score1)
plt.figure()
plt.plot(X_train.squeeze(), Y_train, 'o')
plt.plot(X_train.squeeze(), yhat_RF);

Y_cases = normalized_daily_new_cases_MI.iloc[0].values.tolist()
X = np.array(X)
Y = np.array(Y_cases)
z = np.polyfit(X, Y, 1)
p = np.poly1d(z)

plt.plot(X,p(X),"r-")

plt.xlabel('Number of days')
plt.ylabel('New cases MI')
plt.title('Random Forest MI-cases')<jupyter_output><empty_output><jupyter_text>Random forest for the number of deaths<jupyter_code>Y_deaths = normalized_daily_new_deaths_MI.iloc[0].values.tolist()
X_deaths_train = np.array(X)
Y_deaths_train = np.array(Y_deaths)
Kdeaths_test = np.array(K)

X_deaths_train = X_deaths_train[:, np.newaxis]
Y_deaths_train = Y_deaths_train[:, np.newaxis]
Kdeaths_test = Kdeaths_test[:, np.newaxis]


model1 = RandomForestRegressor()
model1.fit(X_deaths_train, Y_deaths_train)

yhat_RF = model1.predict(X_deaths_train)
yhat_RF1 = model1.predict(Kdeaths_test)

print("The RMSE:",np.sqrt(metrics.mean_squared_error(Y_deaths_train, yhat_RF)))
score1 = r2_score(Y_deaths_train, yhat_RF)
print("R^2: ", score1)
plt.figure()
plt.plot(X_train.squeeze(), Y_train, 'o')
plt.plot(X_train.squeeze(), yhat_RF);

Y_cases = normalized_daily_new_deaths_MI.iloc[0].values.tolist()
X = np.array(X)
Y = np.array(Y_deaths)
z = np.polyfit(X, Y, 1)
p = np.poly1d(z)
plt.plot(X,p(X),"r-")

plt.xlabel('Number of days')
plt.ylabel('New deaths MI')
plt.title('Random Forest MI-deaths')<jupyter_output><empty_output><jupyter_text>Arima<jupyter_code>plt.plot(X_cases, Y_cases)<jupyter_output><empty_output><jupyter_text>Our data seems stationary because the number of daily new cases and deaths that are being used . Moreover, from the graph above we can see that there is no trend: it is neither increasing or decreasing over time. Therefore ARIMA model can be applied to the data.<jupyter_code>from statsmodels.tsa.arima_model import ARIMA 
Xcases_trainD = np.array(X)
Xcases_trainD = Xcases_trainD[:, np.newaxis]
Ycases_trainD = np.array(Y_cases)
Y_cases = pd.DataFrame(normalized_daily_new_cases_MI.iloc[0]) 
Ycases_trainD = np.array(Y_cases)

xtraining, xtest = Xcases_trainD[:196,:], Xcases_trainD[196:,:]
ytraining, ytest = Ycases_trainD[:196,:], Ycases_trainD[196:,:]

arima_model= ARIMA(ytraining, order=(3,1,1))
ari= arima_model.fit()
prediction= ari.forecast(steps=50)[0]
print("The predicted values for cases:", prediction)
pred= [i for i in range(196,246)]
plt.plot(xtest, ytest)
plt.plot(pred, prediction, "red")

plt.xlabel('Number of days')
plt.ylabel('New cases MI')
plt.title('ARIMA MI-cases')
print("The RMSE:", np.sqrt(metrics.mean_squared_error(ytest, prediction)))
score = r2_score(ytest, prediction)
print("The R^2:", score)<jupyter_output>The RMSE: 280.3618737750299
The R^2: -0.273849141616582<jupyter_text>ARIMA for number of deaths<jupyter_code>from statsmodels.tsa.arima_model import ARIMA 
Xcases_trainD = np.array(X)
Xcases_trainD = Xcases_trainD[:, np.newaxis]
Y_death = pd.DataFrame(normalized_daily_new_deaths_MI.iloc[0]) 
Ycases_trainD = np.array(Y_death)
xtraining, xtest = Xcases_trainD[:196,:], Xcases_trainD[196:,:]
ytraining, ytest = Ycases_trainD[:196,:], Ycases_trainD[196:,:]

arima_model= ARIMA(ytraining, order=(0,1,1))
ari= arima_model.fit()
prediction= ari.forecast(steps=50)[0]
print("The predicted values for deaths:", prediction)
pred= [i for i in range(196,246)]
plt.plot(xtest, ytest)
plt.plot(pred, prediction, "red")

plt.xlabel('Number of days')
plt.ylabel('New deaths MI')
plt.title('ARIMA MI-deaths')
print("The RMSE:", np.sqrt(metrics.mean_squared_error(ytest, prediction)))
score = r2_score(ytest, prediction)
print("The R^2:", score)<jupyter_output>The RMSE: 52.089473017361485
The R^2: -0.2807772099643209<jupyter_text>For the Arima model, the R score is negative for both the number of cases and the number of deaths which means the model is doing worse that the average number of cases. This observation can be reliable because ARIMA model focuses on the moving average of the data, so R^2 is not the metric that matters the most in these cases.  Enrichment dataset   From my enrichment dataset, there were 4 variables that were significant for my analysis so I am using all of the variables to study the spread of Covid in cases and deaths. To do so, I basically tried to observe the relationship between each variable and the total number of new cases/deaths  on the last day. If there is a relationship then it means that this variable contributes to the spread of the virus.<jupyter_code>employment.head()
group_by_States = employment.groupby(['State']).sum()
group_by_States
states_empc= (group_by_States.loc[:, group_by_States.columns.str.endswith("_x")])
states_empd= (group_by_States.loc[:, group_by_States.columns.str.endswith("_y")])

newcases_MI_emp = states_empc.diff(axis=1)
newcases_MI_emp=newcases_MI_emp.loc[: ,'3/6/20_x':]
newcases_MI_emp.columns = newcases_MI_emp.columns.str.replace('_x', '')
newcases_MI_emp

newdeaths_MI_emp = states_empd.diff(axis=1)
newdeaths_MI_emp=newdeaths_MI_emp.loc[: ,'3/6/20_y':]
newdeaths_MI_emp.columns = newdeaths_MI_emp.columns.str.replace('_y', '')
newdeaths_MI_emp

stateCases = newcases_MI_emp[20:30:]
stateDeaths= newdeaths_MI_emp[20:30:]
stateDeaths
emp_level= group_by_States[["Establishment Count","January Employment","February Employment", "March Employment"]]
empState= emp_level[20:30]
result_cases = pd.concat([stateCases,empState], axis=1)
some= result_cases.loc[:, '9/14/20':]
some= some.rename(columns={'9/14/20': 'last','Establishment Count': 'Count','January Employment':'Jan','February Employment': 'Feb',
                          'March Employment': 'March'})
some

lm = smf.ols(formula='last ~ Count + Jan + Feb ', data=some).fit()

lm1 = smf.ols(formula='last ~ Count + Jan + Feb +March ', data=some).fit()
# print the coefficients
#print(lm.params)
#print(lm1.params)
print("The p-values are:", "\n",lm1.pvalues)
#model performance

print(lm1.rsquared)
lm1.conf_int()
lm.summary()<jupyter_output>/opt/anaconda3/envs/opencv/lib/python3.8/site-packages/scipy/stats/stats.py:1603: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=10
  warnings.warn("kurtosistest only valid for n>=20 ... continuing "<jupyter_text>Count: 0.054413,  Jan: 0.040547, Feb: 0.018370 ,  March: 0.018533 The p-values for all the variables from the enrichment dataset are less than 0.05 which indicates that there is a relationship between the employment level and the the number of covid19 cases. This is a confirmation as in stage II we have calculated the correlation coefficient and they were really high(more than 0.9). Therefore all these variables are strongly correlated to the number of covid19 cases. They contribute to the spread of the virus<jupyter_code>emp_level= group_by_States[["Establishment Count","January Employment","February Employment", "March Employment"]]
empState= emp_level[20:30]
result_deaths = pd.concat([stateDeaths,empState], axis=1)
some1= result_deaths.loc[:, '9/14/20':]
some1= some1.rename(columns={'9/14/20': 'last','Establishment Count': 'Count','January Employment':'Jan','February Employment': 'Feb',
                          'March Employment': 'March'})
some1


lm = smf.ols(formula='last ~ Count + Jan + Feb +March ', data=some1).fit()
# print the coefficients
print("The p-values are:", "\n",lm.pvalues)
#model performance

print(lm.rsquared)
lm.conf_int()
lm.summary()<jupyter_output>/opt/anaconda3/envs/opencv/lib/python3.8/site-packages/scipy/stats/stats.py:1603: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=10
  warnings.warn("kurtosistest only valid for n>=20 ... continuing "
{'fraction_non_alphanumeric': 0.12948141131896757, 'fraction_numerical': 0.024627042386928723, 'mean_word_length': 6.253864651322569, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 79, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '176475', 'license': 'MIT', 'n_tokens_mistral': 8506, 'n_tokens_neox': 7965, 'n_words': 1748, 'path': 'src/Stage_III/Nadia/Stage3.ipynb', 'repo_name': 'HarinB4/Covid-19DataProject', 'size': '21115'}