<jupyter_start><jupyter_text>Part - 2: COVID-19 Time Series Analysis and Prediction using ML.Net framework  COVID-19- As per [Wiki](https://en.wikipedia.org/wiki/Coronavirus_disease_2019) **Coronavirus disease 2019** (**COVID-19**) is an infectious disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The disease was first identified in 2019 in Wuhan, the capital of China's Hubei province, and has since spread globally, resulting in the ongoing 2019–20 coronavirus pandemic.- The virus had caused a pandemic across the globe and spreading/affecting most of the nations. - The purpose of notebook is to visualize the trends of virus spread in various countries and explore features present in ML.Net such as DataFrame.  Acknowledgement- [Johns Hopkins CSSE](https://github.com/CSSEGISandData/COVID-19/raw/master/csse_covid_19_data) for dataset- [COVID-19 data visualization](https://www.kaggle.com/akshaysb/covid-19-data-visualization) by <NAME>  Dataset- [2019 Novel Coronavirus COVID-19 (2019-nCoV) Data Repository by Johns Hopkins CSSE - Time Series](https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_time_series).  Introduction This is **Part-2** of our analysis on the COVID-19 dataset provided by Johns Hopkins CSSE. In [**Part-1**](https://github.com/praveenraghuvanshi1512/TechnicalSessions/tree/31052020-virtualmlnet/31052020-virtualmlnet/src/part-1), I did data analysis on the dataset and created some tables and plots for getting insights from it. In Part-2, I'll focus on applying machine learning for making a prediction using time-series API's provided by ML.Net framework. I'll be building a model from scratch on the number of confirmed cases and predicting for the next 7 days. Later on, I'll plot these numbers for better visualization.[**ML.Net**](https://dotnet.microsoft.com/apps/machinelearning-ai/ml-dotnet) is a cross-platform framework from Microsoft for developing Machine learning models in the .Net ecosystem. It allows .Net developers to solve business problems using machine learning algorithms leveraging their preferred language such as C/F. It's highly scalable and used within Microsoft in many of its products such as Bing, Powerpoint, etc.**Disclaimer**: This is an exercise to explore different features present in ML.Net. The actual and predicted numbers might vary due to several factors such as size and features in a dataset.  SummaryBelow is the summary of steps we'll be performing1. Define application level items    - Nuget packages    - Namespaces    - Constants     2. Utility Functions    - Formatters    3. Dataset and Transformations    - Actual from [Johns Hopkins CSSE](https://github.com/CSSEGISandData/COVID-19/blob/master/csse_covid_19_data/csse_covid_19_time_series)    - Transformed [time_series_covid19_confirmed_global_transposed.csv](time_series_covid19_confirmed_global_transposed.csv)    4. Data Classes    - ConfirmedData : Provides a map between columns in a dataset    - ConfirmedForecast : Holds predicted values5. Data Analysis    - Visualize Data using DataFrame API    - Display Top 10 Rows - dataframe.Head(10)    - Display Last 10 Rows - dataframe.Tail(10)    - Display Dataset Statistics - dataframe.Description()    - Plot of TotalConfimed cases vs Date6. Load Data - MLContext7. ML Pipeline8. Train Model9. Prediction/Forecasting10. Prediction Visualization11. Prediction Analysis12. Conclusion**Note** : Graphs/Plots may not render in GitHub due to secutiry reasons, however if you run this notebook locally/binder they will render.<jupyter_code>#!about<jupyter_output><empty_output><jupyter_text>1. Define Application wide Items  Nuget Packages<jupyter_code>// ML.NET Nuget packages installation
#r "nuget:Microsoft.ML"
#r "nuget:Microsoft.ML.TimeSeries"
#r "nuget:Microsoft.Data.Analysis"

// Install XPlot package
#r "nuget:XPlot.Plotly"<jupyter_output><empty_output><jupyter_text>Namespaces<jupyter_code>using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.Data.Analysis;
using Microsoft.ML.Transforms.TimeSeries;
using Microsoft.AspNetCore.Html;
using XPlot.Plotly;<jupyter_output><empty_output><jupyter_text>Constants<jupyter_code>const string CONFIRMED_DATASET_FILE = "time_series_covid19_confirmed_global_transposed.csv";

// Forecast API
const int WINDOW_SIZE = 5;
const int SERIES_LENGTH = 10;
const int TRAIN_SIZE = 100;
const int HORIZON = 7;

// Dataset
const int DEFAULT_ROW_COUNT = 10;
const string TOTAL_CONFIRMED_COLUMN = "TotalConfirmed";
const string DATE_COLUMN = "Date";<jupyter_output><empty_output><jupyter_text>2. Utility Functions - TBR  FormattersBy default the output of DataFrame is not proper and in order to display it as a table, we need to have a custom formatter implemented as shown in next cell.<jupyter_code>Formatter<DataFrame>.Register((df, writer) =>
{
    var headers = new List<IHtmlContent>();
    headers.Add(th(i("index")));
    headers.AddRange(df.Columns.Select(c => (IHtmlContent) th(c.Name)));
    var rows = new List<List<IHtmlContent>>();
    var take = DEFAULT_ROW_COUNT;
    for (var i = 0; i < Math.Min(take, df.Rows.Count); i++)
    {
        var cells = new List<IHtmlContent>();
        cells.Add(td(i));
        foreach (var obj in df.Rows[i])
        {
            cells.Add(td(obj));
        }
        rows.Add(cells);
    }

    var t = table(
        thead(
            headers),
        tbody(
            rows.Select(
                r => tr(r))));

    writer.Write(t);
}, "text/html");<jupyter_output><empty_output><jupyter_text>3. Dataset and Transformations  Download Dataset- Actual Dataset: [Johns Hopkins CSSE](https://github.com/CSSEGISandData/COVID-19/blob/master/csse_covid_19_data/csse_covid_19_time_series)- Transformed Dataset: [time_series_covid19_confirmed_global_transposed.csv](time_series_covid19_confirmed_global_transposed.csv)I'll be using COVID-19 time series dataset from [Johns Hopkins CSSE](https://github.com/CSSEGISandData/COVID-19/blob/master/csse_covid_19_data/csse_covid_19_time_series) and will be performing predictions using **time_series_covid19_confirmed_global.csv** file.The data present in these files have name of the countries as Rows and dates as columns which makes it difficult to map to our classes while loading data from csv. Also, it contains data per country wise. In order to keep things simple I'll work with global count of COVID-19 cases and not specific country.I have done few transformations to the dataset as below and created transformed csv's- Sum cases from all the countries for a specific date- Just have two rows with Date and Total - Applied transformation to the csv for converting Rows into Columns and vice-versa. [Refer](https://support.office.com/en-us/article/transpose-rotate-data-from-rows-to-columns-or-vice-versa-3419f2e3-beab-4318-aae5-d0f862209744) for transformation.- Below transposed files have been saved in the current github directory. There is no change in dataset. The files have data till 05-27-2020    - [time_series_covid19_confirmed_global_transposed.csv](time_series_covid19_confirmed_global_transposed.csv) : Columns - **Date, TotalConfirmed**  Before transformation  After transformation  4. Data Classes Now, we need to create few data structures to map to columns within our dataset.  Confirmed cases<jupyter_code>/// <summary>
/// Represent data for confirmed cases with a mapping to columns in a dataset
/// </summary>
public class ConfirmedData
{
    /// <summary>
    /// Date of confirmed case
    /// </summary>
    [LoadColumn(0)]
    public DateTime Date;

    /// <summary>
    /// Total no of confirmed cases on a particular date
    /// </summary>
    [LoadColumn(1)]
    public float TotalConfirmed;
}
/// <summary>
/// Prediction/Forecast for Confirmed cases
/// </summary>
internal class ConfirmedForecast
{
    /// <summary>
    /// No of predicted confirmed cases for multiple days
    /// </summary>
    public float[] Forecast { get; set; }
}<jupyter_output><empty_output><jupyter_text>5. Data Analysis For loading data from csv, first we need to create MLContext that acts as a starting point for creating a machine learning model in ML.Net. Few things to note- Set hasHeader as true as our dataset has header- Add separatorChar to ',' as its a csv  Visualize Data - DataFrame<jupyter_code>var predictedDf = DataFrame.LoadCsv(CONFIRMED_DATASET_FILE);
predictedDf.Head(DEFAULT_ROW_COUNT)
predictedDf.Tail(DEFAULT_ROW_COUNT)
predictedDf.Description()<jupyter_output><empty_output><jupyter_text>Number of Confirmed cases over Time<jupyter_code>// Number of confirmed cases over time
var totalConfirmedDateColumn = predictedDf.Columns[DATE_COLUMN];
var totalConfirmedColumn = predictedDf.Columns[TOTAL_CONFIRMED_COLUMN];

var dates = new List<string>();
var totalConfirmedCases = new List<string>();
for (int index = 0; index < totalConfirmedDateColumn.Length; index++)
{
    dates.Add(totalConfirmedDateColumn[index].ToString());
    totalConfirmedCases.Add(totalConfirmedColumn[index].ToString());
}
var title = "Number of Confirmed Cases over Time";
var confirmedTimeGraph = new Graph.Scattergl()
    {
        x = dates.ToArray(),
        y = totalConfirmedCases.ToArray(),
        mode = "lines+markers"
    };
    


var chart = Chart.Plot(confirmedTimeGraph);
chart.WithTitle(title);
display(chart);<jupyter_output><empty_output><jupyter_text>**Analysis**- Duration: 1/22/2020 through 5/27/2020- Total records: 127- Case on first day: 555- Case on last day: 5691790- No of confirmed cases was low in the beginning, there was first jump around 2/12/2020 and an exponential jump around 3/22/2020.- Cases have been increasing at an alarming rate in the past two months.  6. Load Data - MLContext<jupyter_code>var context = new MLContext();
var data = context.Data.LoadFromTextFile<ConfirmedData>(CONFIRMED_DATASET_FILE, hasHeader: true, separatorChar: ',');<jupyter_output><empty_output><jupyter_text>7. ML Pipeline For creating ML Pipeline for a time-series analysis, we'll use [Single Spectrum Analysis](https://en.wikipedia.org/wiki/Singular_spectrum_analysis). ML.Net provides built in API for same, more details could be found at [TimeSeriesCatalog.ForecastBySsa](https://docs.microsoft.com/en-us/dotnet/api/microsoft.ml.timeseriescatalog.forecastbyssa?view=ml-dotnet)<jupyter_code>var pipeline = context.Forecasting.ForecastBySsa(
                nameof(ConfirmedForecast.Forecast),
                nameof(ConfirmedData.TotalConfirmed),
                WINDOW_SIZE, 
                SERIES_LENGTH,
                TRAIN_SIZE,
                HORIZON);<jupyter_output><empty_output><jupyter_text>8. Train Model We are ready with our pipeline and ready to train the model<jupyter_code>var model = pipeline.Fit(data);<jupyter_output><empty_output><jupyter_text>9. Prediction/Forecasting - 7 days Our model is trained and we need to do prediction for next 7(Horizon) days.Time-series provides its own engine for making prediction which is similar to PredictionEngine present in ML.Net. Predicted values show an increasing trend which is in alignment with recent past values.<jupyter_code>var forecastingEngine = model.CreateTimeSeriesEngine<ConfirmedData, ConfirmedForecast>(context);
var forecasts = forecastingEngine.Predict();
display(forecasts.Forecast.Select(x => (int) x))<jupyter_output><empty_output><jupyter_text>10. Prediction Visualization<jupyter_code>var lastDate = DateTime.Parse(dates.LastOrDefault());
var predictionStartDate = lastDate.AddDays(1);

for (int index = 0; index < HORIZON; index++)
{
    dates.Add(lastDate.AddDays(index + 1).ToShortDateString());
    totalConfirmedCases.Add(forecasts.Forecast[index].ToString());
}
var title = "Number of Confirmed Cases over Time";
var layout = new Layout.Layout();
layout.shapes = new List<Graph.Shape>
{
    new Graph.Shape
    {
        x0 = predictionStartDate.ToShortDateString(),
        x1 = predictionStartDate.ToShortDateString(),
        y0 = "0",
        y1 = "1",
        xref = 'x',
        yref = "paper",
        line = new Graph.Line() {color = "red", width = 2}
    }
};

var chart1 = Chart.Plot(
new [] 
    {
        new Graph.Scattergl()
        {
            x = dates.ToArray(),
            y = totalConfirmedCases.ToArray(),
            mode = "lines+markers"
        }
    },
    layout
);

chart1.WithTitle(title);
display(chart1);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09328298248433231, 'fraction_numerical': 0.019926080668487868, 'mean_word_length': 4.414093083949544, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 78, 'https://': 12, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '4314', 'license': 'MIT', 'n_tokens_mistral': 3891, 'n_tokens_neox': 3623, 'n_words': 1153, 'path': 'part-2/Part-2-COVID-19-mlnet-prediction.ipynb', 'repo_name': 'praveenraghuvanshi1512/covid-19', 'size': '12449'}