<jupyter_start><jupyter_text>EDA  About the Data 1. Contains Daily Reports of Number of Cases and Fatalities for countries.2. [Missing Data]Contains some entries with Province/State Information Missing - Dropped.3. Contains latitude and longitude for entries, Can Plot on map.4. Date - 22nd Feb to 23nd March.5. Country/Region - 163<jupyter_code># Dataset Dimesnions
print('Train shape', df_train.shape)
print('Test shape', df_test.shape)
# Missing/Null Values
print('\nTrain Missing\n', df_train.isnull().sum())
print('\nTest Missing\n', df_test.isnull().sum())<jupyter_output>Train shape (17892, 8)
Test shape (12212, 6)

Train Missing
 Id                   0
Province/State    9702
Country/Region       0
Lat                  0
Long                 0
Date                 0
ConfirmedCases       0
Fatalities           0
dtype: int64

Test Missing
 ForecastId           0
Province/State    6622
Country/Region       0
Lat                  0
Long                 0
Date                 0
dtype: int64<jupyter_text>Unique countries in the dataset<jupyter_code>lst = df_train['Country/Region'].unique()
print('Total_Countries\n:', len(lst))
for i in lst:
    print(i)<jupyter_output>Total_Countries
: 163
Afghanistan
Albania
Algeria
Andorra
Antigua and Barbuda
Argentina
Armenia
Aruba
Australia
Austria
Azerbaijan
Bahrain
Bangladesh
Barbados
Belarus
Belgium
Benin
Bhutan
Bolivia
Bosnia and Herzegovina
Brazil
Brunei
Bulgaria
Burkina Faso
Cambodia
Cameroon
Canada
Central African Republic
Chile
China
Colombia
Congo (Brazzaville)
Congo (Kinshasa)
Costa Rica
Cote d'Ivoire
Croatia
Cruise Ship
Cuba
Cyprus
Czechia
Denmark
Djibouti
Dominican Republic
Ecuador
Egypt
Equatorial Guinea
Estonia
Eswatini
Ethiopia
Finland
France
French Guiana
Gabon
Gambia, The
Georgia
Germany
Ghana
Greece
Greenland
Guadeloupe
Guam
Guatemala
Guernsey
Guinea
Guyana
Holy See
Honduras
Hungary
Iceland
India
Indonesia
Iran
Iraq
Ireland
Israel
Italy
Jamaica
Japan
Jersey
Jordan
Kazakhstan
Kenya
Korea, South
Kosovo
Kuwait
Kyrgyzstan
Latvia
Lebanon
Liberia
Liechtenstein
Lithuania
Luxembourg
Malaysia
Maldives
Malta
Martinique
Mauritania
Mauritius
Mayotte
Mexico
Moldova
Monaco
Mongolia
Montenegro
Morocco
Namibia[...]<jupyter_text>Date Range for the Dataset<jupyter_code>print(df_train['Date'].min(), ' - ', df_train['Date'].max())<jupyter_output>2020-01-22  -  2020-03-24<jupyter_text>Checking Daily Worldwide Confirmed Cases and Fatalities<jupyter_code># GroupBy syntax (columns to group by in list)[Columns to aggregate, apply function to] . aggregation functions on it 
train_cases_conf = df_train.groupby(['Date'])['ConfirmedCases'].sum()
train_cases_conf
train_cases_conf.plot(figsize = (10,8), title = 'Worldwide Confirmed Cases')
train_fatal = df_train.groupby(['Date'])['Fatalities'].sum()
train_fatal
train_fatal.plot(figsize = (10,8), title = 'Worldwide Fatalaties')<jupyter_output><empty_output><jupyter_text>Check Confirmed cases and fatalities for a country scale = "linear", "log"<jupyter_code>def country_stats(country, df):
    country_filt = (df['Country/Region'] == country)
    df_cases = df.loc[country_filt].groupby(['Date'])['ConfirmedCases'].sum()
    df_fatal = df.loc[country_filt].groupby(['Date'])['Fatalities'].sum()
    fig, axes = plt.subplots(nrows = 2, ncols= 1, figsize=(15,15))
    df_cases.plot(ax = axes[0])
    df_fatal.plot(ax = axes[1])
    
country_stats('US', df_train)<jupyter_output><empty_output><jupyter_text>Fatalities and Confirmed Cases by Country (Log Scale)<jupyter_code># grouping using same Country filter to get fatalities on each date (grouped by date)
# groupby([list of columns to groupby]) [which columns to apply aggregate fx to ]. (aggregate function)

def country_stats_log(country, df):
    count_filt =(df_train['Country/Region'] == country)
    df_count_case = df_train.loc[count_filt].groupby(['Date'])['ConfirmedCases'].sum()
    df_count_fatal = df_train.loc[count_filt].groupby(['Date'])['Fatalities'].sum()
    plt.figure(figsize=(15,10))
    plt.axes(yscale = 'log')
    plt.plot(df_count_case.index, df_count_case.tolist(), 'b', label = country +' Total Confirmed Cases')
    plt.plot(df_count_fatal.index, df_count_fatal.tolist(), 'r', label = country +' Total Fatalities')
    plt.title(country +' COVID Cases and Fatalities (Log Scale)')
    plt.legend()
    

country_stats_log('US', df_train)<jupyter_output><empty_output><jupyter_text>Most Affected Countries<jupyter_code># as_index = False to not make the grouping column the index, creates a df here instead of series, preserves
# Confirmedcases column

train_case_country = df_train.groupby(['Country/Region'], as_index=False)['ConfirmedCases'].max()

# Sorting by number of cases
train_case_country.sort_values('ConfirmedCases', ascending=False, inplace = True)
train_case_country
plt.figure(figsize=(8,6))
plt.bar(train_case_country['Country/Region'][:5], train_case_country['ConfirmedCases'][:5], color = ['red', 'yellow','black','blue','green'])<jupyter_output><empty_output><jupyter_text>No. of Cases on final date(2020/03/23), (Not Increase, Cumulative)<jupyter_code># Confirmed Cases till a particular day by country

def case_day_country (Date, df):
    df = df.groupby(['Country/Region', 'Date'], as_index = False)['ConfirmedCases'].sum()
    date_filter = (df['Date'] == Date)
    df = df.loc[date_filter]
    df.sort_values('ConfirmedCases', ascending = False, inplace = True)
    sns.catplot(x = 'Country/Region', y = 'ConfirmedCases' , data = df.head(10), height=5,aspect=3, kind = 'bar')
    
    
case_day_country('2020-03-23', df_train)<jupyter_output><empty_output><jupyter_text>Data Wrangling/ Pre-processing/ Cleaning 1. Identifying and Handling missing values.2. Data Formating.3. Data Normalization(centering and scaling).4. Data bining.5. Turning categorical values into numerical values.  Need to Exclude Leaky Data, the same Dates are in both train and test set.1. First convert object to python datetime type Using pd.to_datetime() Check Getting converted to float, because haven't converted to date before comparison, still object.<jupyter_code>df_train.Date = pd.to_datetime(df_train['Date'])
print(df_train['Date'].max())
print(df_test['Date'].min())<jupyter_output>2020-03-24 00:00:00
2020-03-12<jupyter_text>Truncate df_train by date < df_test['Date'].min()<jupyter_code>date_filter = df_train['Date'] < df_test['Date'].min()
df_train = df_train.loc[date_filter]
# Dropping ID and getting rid of Province/State with NULL values 
df_train.info()
# lets get Cumulative sum of ConfirmedCases and Fatalities for each country on each data (same as original data)
# Doing to create copy without ID and 

train_country_date = df_train.groupby(['Country/Region', 'Date', 'Lat', 'Long'], as_index=False)['ConfirmedCases', 'Fatalities'].sum()
print(train_country_date.info())
print(train_country_date.isnull().sum())<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 14200 entries, 0 to 14199
Data columns (total 6 columns):
 #   Column          Non-Null Count  Dtype         
---  ------          --------------  -----         
 0   Country/Region  14200 non-null  object        
 1   Date            14200 non-null  datetime64[ns]
 2   Lat             14200 non-null  float64       
 3   Long            14200 non-null  float64       
 4   ConfirmedCases  14200 non-null  float64       
 5   Fatalities      14200 non-null  float64       
dtypes: datetime64[ns](1), float64(4), object(1)
memory usage: 776.6+ KB
None
Country/Region    0
Date              0
Lat               0
Long              0
ConfirmedCases    0
Fatalities        0
dtype: int64<jupyter_text>Feature EngineeringSplitting Date into day, month, day of week. Check if Date is in python datetime format. Else, convert object to python datetime type Using pd.to_datetime()<jupyter_code>train_country_date.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 14200 entries, 0 to 14199
Data columns (total 6 columns):
 #   Column          Non-Null Count  Dtype         
---  ------          --------------  -----         
 0   Country/Region  14200 non-null  object        
 1   Date            14200 non-null  datetime64[ns]
 2   Lat             14200 non-null  float64       
 3   Long            14200 non-null  float64       
 4   ConfirmedCases  14200 non-null  float64       
 5   Fatalities      14200 non-null  float64       
dtypes: datetime64[ns](1), float64(4), object(1)
memory usage: 776.6+ KB<jupyter_text>Using Pandas Series.dt.monthThe month as January=1, December=12.<jupyter_code># Adding day, month, day of week columns 

train_country_date['Month'] = train_country_date['Date'].dt.month
train_country_date['Day'] = train_country_date['Date'].dt.day
train_country_date['Day_Week'] = train_country_date['Date'].dt.dayofweek
train_country_date['quarter'] = train_country_date['Date'].dt.quarter
train_country_date['dayofyear'] = train_country_date['Date'].dt.dayofyear
train_country_date['weekofyear'] = train_country_date['Date'].dt.weekofyear
train_country_date.head()
train_country_date.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 14200 entries, 0 to 14199
Data columns (total 12 columns):
 #   Column          Non-Null Count  Dtype         
---  ------          --------------  -----         
 0   Country/Region  14200 non-null  object        
 1   Date            14200 non-null  datetime64[ns]
 2   Lat             14200 non-null  float64       
 3   Long            14200 non-null  float64       
 4   ConfirmedCases  14200 non-null  float64       
 5   Fatalities      14200 non-null  float64       
 6   Month           14200 non-null  int64         
 7   Day             14200 non-null  int64         
 8   Day_Week        14200 non-null  int64         
 9   quarter         14200 non-null  int64         
 10  dayofyear       14200 non-null  int64         
 11  weekofyear      14200 non-null  int64         
dtypes: datetime64[ns](1), float64(4), int64(6), object(1)
memory usage: 1.4+ MB<jupyter_text>Same Feature Engineering for Test Set<jupyter_code># First drop Province/State
df_test.drop('Province/State', axis = 1, inplace = True)

# Converting Date Object to Datetime type

df_test.Date = pd.to_datetime(df_test['Date'])
df_test.head(2)
# adding Month, DAy, Day_week columns Using Pandas Series.dt.month

df_test['Month'] = df_test['Date'].dt.month
df_test['Day'] = df_test['Date'].dt.day
df_test['Day_Week'] = df_test['Date'].dt.dayofweek
df_test['quarter'] = df_test['Date'].dt.quarter
df_test['dayofyear'] = df_test['Date'].dt.dayofyear
df_test['weekofyear'] = df_test['Date'].dt.weekofyear
df_test.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 12212 entries, 0 to 12211
Data columns (total 11 columns):
 #   Column          Non-Null Count  Dtype         
---  ------          --------------  -----         
 0   ForecastId      12212 non-null  int64         
 1   Country/Region  12212 non-null  object        
 2   Lat             12212 non-null  float64       
 3   Long            12212 non-null  float64       
 4   Date            12212 non-null  datetime64[ns]
 5   Month           12212 non-null  int64         
 6   Day             12212 non-null  int64         
 7   Day_Week        12212 non-null  int64         
 8   quarter         12212 non-null  int64         
 9   dayofyear       12212 non-null  int64         
 10  weekofyear      12212 non-null  int64         
dtypes: datetime64[ns](1), float64(2), int64(7), object(1)
memory usage: 1.0+ MB<jupyter_text>Councatenating Train-Test to Label encode Country/Region Categorical Variable.1. Make copy of train data without Confirmed Cases and Fatalities. Index - 0 to 176082. Concatenate train, test.3. Label Encode Countries.4. Add back Cofirmed Cases, Fatalities columns to clean_train_data.5. Modelling6. Saving Predicted Values with ForecastID<jupyter_code># train_country_date
# df_test
# Lets select the Common Labels and concatenate.

labels = ['Country/Region', 'Lat', 'Long', 'Date', 'Month', 'Day', 'Day_Week','quarter', 'dayofyear', 'weekofyear']

df_train_clean = train_country_date[labels]
df_test_clean = df_test[labels]

data_clean = pd.concat([df_train_clean, df_test_clean], axis = 0)
data_clean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 26412 entries, 0 to 12211
Data columns (total 10 columns):
 #   Column          Non-Null Count  Dtype         
---  ------          --------------  -----         
 0   Country/Region  26412 non-null  object        
 1   Lat             26412 non-null  float64       
 2   Long            26412 non-null  float64       
 3   Date            26412 non-null  datetime64[ns]
 4   Month           26412 non-null  int64         
 5   Day             26412 non-null  int64         
 6   Day_Week        26412 non-null  int64         
 7   quarter         26412 non-null  int64         
 8   dayofyear       26412 non-null  int64         
 9   weekofyear      26412 non-null  int64         
dtypes: datetime64[ns](1), float64(2), int64(6), object(1)
memory usage: 2.2+ MB<jupyter_text>Preparing Data For Models - LabelEncode Country<jupyter_code>from sklearn.preprocessing import LabelEncoder
# Label Encoder for Countries 

enc = LabelEncoder()
data_clean['Country'] = enc.fit_transform(data_clean['Country/Region'])
data_clean
# Dropping Country/Region and Date

data_clean.drop(['Country/Region', 'Date'], axis = 1, inplace=True)<jupyter_output><empty_output><jupyter_text>Splitting Back into Train and Test<jupyter_code>index_split = df_train.shape[0]
data_train_clean = data_clean[:index_split]
data_test_clean = data_clean[index_split:]<jupyter_output><empty_output><jupyter_text>Adding Back Confirmed Cases and FatalitiesUsing original df_train, check shape is same, head, tail have same values. ORDER NEEDS TO BE SAME.<jupyter_code>data_train_clean.tail(5)<jupyter_output><empty_output><jupyter_text>Creating Features and Two Labels<jupyter_code>x = data_train_clean[['Lat', 'Long', 'Month', 'Day', 'Day_Week','quarter', 'dayofyear', 'weekofyear', 'Country']]
y_case = df_train['ConfirmedCases']
y_fatal = df_train['Fatalities']<jupyter_output><empty_output><jupyter_text>Train-Test Split - Confirmed Cases<jupyter_code>from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(x, y_case, test_size = 0.3, random_state = 42)<jupyter_output><empty_output><jupyter_text>Train-Test Split - Fatalities<jupyter_code>from sklearn.model_selection import train_test_split

x_train_fatal, x_test_fatal, y_train_fatal, y_test_fatal = train_test_split(x, y_fatal, test_size = 0.3, random_state = 42)<jupyter_output><empty_output><jupyter_text>Modeling - Regression Problem Using features Country/Region, Lat, Long, Month, Day, Day_week, quarter, dayofyear, weekofyear.To predict ConfirmedCases, Fatalities. To predict 2 Different Target Variables, Train two classifiers, one for each.  Modelling  RandomForest Regressor<jupyter_code>from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error<jupyter_output><empty_output><jupyter_text>For ConfirmedCases<jupyter_code>rf = RandomForestRegressor(n_estimators =100)
rf.fit(x_train, y_train.values.ravel())
rf.score(x_train, y_train)
rf.score(x_test, y_test)
# Predicted Values and MSE
y_pred_train = rf.predict(x_train)
print(mean_squared_error(y_train, y_pred_train))
# Training on entire set and predict values.

rf.fit(x, y_case.values.ravel())
# Predicted ConfirmedCases
rf_pred_case = rf.predict(data_test_clean)
plt.figure(figsize=(15,8))
plt.plot(rf_pred_case)<jupyter_output><empty_output><jupyter_text>For Fatalities<jupyter_code>rf.fit(x, y_fatal.values.ravel())
rf_pred_fatal = rf.predict(data_test_clean)
plt.figure(figsize=(20,8))
plt.plot(rf_pred_fatal)
# Saving to Submission.csv

submission = pd.read_csv(submit_kaggle)
submission['ConfirmedCases'] = rf_pred_case
submission['Fatalities'] = rf_pred_fatal

submission.to_csv('submission.csv', index = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10781990521327015, 'fraction_numerical': 0.04153155400349214, 'mean_word_length': 3.8144701290903633, 'pattern_counts': {'":': 0, '<': 91, '<?xml version=': 0, '>': 89, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '209397', 'license': 'Apache-2.0', 'n_tokens_mistral': 5887, 'n_tokens_neox': 5296, 'n_words': 1320, 'path': 'DongminShin/279_COVID19 Global Forecasting (Week 1)/solution.ipynb', 'repo_name': 'seculayer/AutoAPE-challenge1', 'size': '16036'}