<jupyter_start><jupyter_text>CRISP-DM![CRISP-DM](../reports/figures/CRISP_DM.png)  Modelling- Spread- We want to focus on modelling how the virus spreads<jupyter_code># Utility plotting function
def quick_plot(x_data, y_data, type="log", xslider=False):
    """ Basic plot for rapid evaluation of time series data

    Parameters:
    ----------
    x_data: array
        array of numbers or datetime objects
    y_data: pandas dataframe
        matrix to plot, each column is plotted as a trace
        the column name would be used as legend entry for the trace 
    type: str
        y-axis scale, 'log' or 'linear'
    xslider: bool
        x-axis slider, True or False

    Returns:
    -------

    """
    # Create figure
    fig= go.Figure()

    # Column list
    column_list= y_data.columns
    # Loop through each column
    for column in column_list:
        # Add a trace
        fig.add_trace(
            go.Scatter(
                x=x_data,
                y=y_data[column],
                mode="lines",
                opacity=0.8,
                line_width=2,
                marker_size=5,
                name= column
            )
        )

    # Set figure layout
    fig.update_layout(
        width=900,
        height=600,
        xaxis_title="Time",
        xaxis={
            "tickangle": -75,
            "nticks": 20,
            "tickfont": dict(size=14, color="#7f7f7f")
        },
        yaxis_title="Quick Plot",
        yaxis={
            "type": type
        }
    )

    # Introduce range slider on x-axis
    if(xslider):
        fig.update_layout(xaxis_rangeslider_visible=True)

    # Display figure
    fig.show()

# Import compact dataset
df_base= pd.read_csv("../data/processed/COVID_flat_small.csv", sep=";")
df_base.tail()
# Try quick plot
quick_plot(df_base.iloc[:, 0], df_base.iloc[:, 1:], type="linear", xslider=True)
# Align time series to coincide at some pre-defined threshold
# This allows us to compare behaviour

threshold= 1000

country_list= df_base.columns.drop("date")
# Temporary list
temp_list=[]

for pos,country in enumerate(country_list):
    # Slice off parts of the dataframe that's above the specified threshold and append to list
    temp_list.append(np.array(df_base[country][df_base[country]>threshold]))
# Push list into DataFrame
df_thres= pd.DataFrame(temp_list)
df_thres.tail()
# Set country names as index
df_thres= df_thres.set_index(country_list)
df_thres.head()
# Transpose
df_thres= df_thres.T
df_thres
# x_data for plot
x_data= np.arange(df_thres.shape[0])

# Quick Plot
quick_plot(x_data, df_thres)<jupyter_output><empty_output><jupyter_text>**Exponential Function**  $N(t, T)= N_0 * 2^{t/T}$  $N_{0}$- base value  t- time  T- base time (i.e. $N(t,2)$ is the doubling rate for every 2 days)<jupyter_code>def doubled_series(N_0, t, T):
    """ Calculate the doubled time series for a specified doubling rate

    Parameters:
    ----------
    N_0: double
        initial value
    t: array Nx1
        time
    T: int
        base time

    Returns:
    -------
    doubling_rate: array Nx1
    """
    return N_0 * np.power(2, (t/T))
# Calculate the doubled series for different doubling rates
d_rates= [10,12,15]

# Will hold output series
df_series= {}

for rate in d_rates:
    df_series["doubling every {0} days".format(rate)]= doubled_series(threshold, x_data, rate)
# Concatenate doubled series with synchronized timeline series
df_sync_with_slopes= pd.concat([pd.DataFrame(df_series), df_thres], axis=1)
# Plot Data with doubled series (which are essentially normalized slopes)
quick_plot(x_data, df_sync_with_slopes)
# Save dataset
df_sync_with_slopes.to_csv("../data/processed/COVID_small_sync_timelines.csv", sep=";")<jupyter_output><empty_output><jupyter_text>Understanding Linear Regression  Nigeria<jupyter_code># Create Linear Regression Model
reg= linear_model.LinearRegression(fit_intercept= False)    
#fit_intercept= False sets the y-axis intercept to zero.
# Setup fitting data
y= np.array(df_base["Nigeria"])
y.shape
# First non-zero index
not_0_idx= np.where(y>0)
not_0_idx
# Truncate data to start from first non-zero index
y= y[not_0_idx[0][0]:]
# scikit-learn takes X matrix with rows as features and columns as samples
nig_row= np.arange(y.shape[0])
X= nig_row.reshape(-1, 1)
#np.reshape(-1,1) infers the row number from the array upon which it is applied<jupyter_output><empty_output><jupyter_text>Linear Regression Fit (Nigeria)<jupyter_code>reg.fit(X,y)
# Inspect Solution
X_hat= X
y_hat= reg.predict(X_hat)
LR_inspect= pd.DataFrame([df_base["date"], df_base["Nigeria"]])
LR_inspect.head()
LR_inspect= LR_inspect.T
LR_inspect.tail()
LR_inspect.set_index= "date"
# Truncate data to start from first non-zero index
LR_inspect= LR_inspect[not_0_idx[0][0]:]
LR_inspect["prediction"]= y_hat
LR_inspect.tail()
# Quick Plot
quick_plot(LR_inspect["date"], LR_inspect.iloc[:,1:], type='linear', xslider=True)<jupyter_output><empty_output><jupyter_text>Data Transformation to Log DomainHere, we transform out data into the Logarithimic domain before applying linear regression.  However, it's imperative to remember that the prediction outcome must be transformed back to linear domain for comparison with original data.<jupyter_code>reg= linear_model.LinearRegression()
# Transform data to log domain
y= np.log(np.array(df_base["Nigeria"]))
# Slice out zero-values
y= y[not_0_idx[0][0]:]
# X data
nig_row= np.arange(y.shape[0])
X= nig_row.reshape(-1, 1)

# Linear Regression
reg.fit(X,y)

# Get estimated values
y_hat= reg.predict(X)

# Convert back to linear domain
y_hat= np.exp(y_hat)

# Replace predicted values in LR_inspect
LR_inspect["prediction"]= y_hat

# Make quick plot
quick_plot(LR_inspect["date"], LR_inspect.iloc[:,1:], type='log', xslider=True)<jupyter_output><empty_output><jupyter_text>Piecewise Linear Regression- Doubling Rate<jupyter_code>def get_rate_via_regression(in_array):
    """ Approximate the doubling time using linear regression.

    3 datapoints are used to approximate the number of days 
    it takes for the number of infected people to double at each point.

    Parameters:
    ----------
    in_array: list/numpy array
        input data

    Returns:
    -------
    doubling_time: double
    """
    
    # Assert output vector is 3 datapoints long
    assert len(in_array)==3

    y= np.array(in_array)
    # Calculate slope using central difference
    # Remeber: scikit-learn takes X matrix with rows as features and columns as samples
    X= np.arange(-1,2).reshape(-1,1)

    # Fit data
    reg.fit(X,y)
    intercept= reg.intercept_
    slope= reg.coef_

    return intercept/slope

def doubling_time_closed(in_array):
    """ Calculate the doubling time in closed form
    """
    y= np.array(in_array)

    return len(y)*np.log(2)/np.log(y[-1]/y[0])<jupyter_output><empty_output><jupyter_text>Doubling Time for Nigeria<jupyter_code># Setup fitting data
fitt_data= df_base["Nigeria"]<jupyter_output><empty_output><jupyter_text>**rolling()** in PandasThis function moves a window across a time series<jupyter_code># Doubling time (apprx)
fitt_db_time= fitt_data.rolling(
    window=3,
    min_periods=3
).apply(get_rate_via_regression, raw=False)
# Doubling time (closed form)
fitt_db_closed= fitt_data.rolling(
    window=3,
    min_periods=3
).apply(doubling_time_closed)
# Quick plot
quick_plot(df_base["date"], pd.DataFrame({"apprx": fitt_db_time, "closed": fitt_db_closed}), type='linear')<jupyter_output><empty_output><jupyter_text>Cleaning up the input data- We want to filter the data in such a way that we do not distort the trend.  Savitzky-Golay Filter<jupyter_code>country_filter_list= []
# Filter input data
for ctry in country_list:
    df_base[ctry + '-filter']= signal.savgol_filter(
        df_base[ctry],
        3, # filtering window
        1, # order of polynomial with which to fit
    )
    # Add to filter list
    country_filter_list.append(ctry + '-filter')

df_base.columns<jupyter_output><empty_output><jupyter_text>Run doubling time estimation on cleaned data<jupyter_code># No of days over which to estimate
days_over= 3

for ctry in country_filter_list:
    df_base[ctry + '-dt']= df_base[ctry].rolling(
        window=days_over,
        min_periods=days_over
    ).apply(get_rate_via_regression, raw=False)
df_base.columns<jupyter_output><empty_output><jupyter_text>Plot doubling rates<jupyter_code># Starting position
start_pos= 100

quick_plot(
    df_base.iloc[start_pos:,0], df_base.iloc[start_pos:, [12, 13,14,15,16]],
    type='linear', xslider=True
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11151288930362767, 'fraction_numerical': 0.01143123760643882, 'mean_word_length': 3.855039637599094, 'pattern_counts': {'":': 6, '<': 45, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '636339', 'license': 'MIT', 'n_tokens_mistral': 3042, 'n_tokens_neox': 2901, 'n_words': 761, 'path': 'notebooks/Modelling_spread.ipynb', 'repo_name': 'Faaizz/codiv_19_analysis', 'size': '8573'}