<jupyter_start><jupyter_text>Imports<jupyter_code>import pandas as pd
import datetime as dt
import numpy as np
import random

from skautobots.date_processing import DateLinear

from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Basic Example  Build Dataset We will start our dataset on January 1st 2015 and generate dates for the next 1500 days. Then we will use the transformer to get the different linear pieces of the date.<jupyter_code>start_date = dt.datetime(year=2015, month=1, day=1)
days_out = 1500
basic_df = pd.DataFrame.from_records(
    [
        {
            'x': x,
            'date': start_date + dt.timedelta(days=x)
        } for x in range(days_out)
    ]
)<jupyter_output><empty_output><jupyter_text>Check the dataframe.<jupyter_code>basic_df.head()<jupyter_output><empty_output><jupyter_text>Load Transformer To create the transformer we pass in the columns we would like to transform, the periods that we would like to capture and whether or not we want the original datetime columns dropped.<jupyter_code>date_transformer = DateLinear(
    cols=['date'], 
    periods=['dayofweek', 'dayofmonth', 'dayofyear', 'year', 'dayofweek', 'monthofyear'],
    drop_base_cols=False
)<jupyter_output><empty_output><jupyter_text>We then use the scikit-learn fit/transform api to transform our data.  Transform Data<jupyter_code>transformed_df = date_transformer.fit_transform(basic_df)
transformed_df.head(40).tail(20)
transformed_df.tail(10)<jupyter_output><empty_output><jupyter_text>Boom we've got all of the individual linear pieces from the date column.  Regression Example  Build Sample Dataset Generate a number of samples and generate a number of y samples based on some of the linear terms of the date. We will use the formula:    $$ 2 * (dayofmonth) + 10 * (year - 2000) - (dayofweek) + noise $$We will use  guassian noise with a mean of 0 and standard deviation of 1.<jupyter_code>noise_mu = 0
noise_sigma = 3
def y_formula(sample_date):
    return (
        (2 * sample_date.day) + 
        10 * (sample_date.year - 2000) - 
        (sample_date.weekday()) + 
        np.random.normal(noise_mu, noise_sigma)
    )<jupyter_output><empty_output><jupyter_text>First we set the number of samples, the start sample date and the max time in the future for the records to be in.<jupyter_code>number_of_samples = 10_000
max_days_out = 1200
start = dt.datetime(year=2015, month=1, day=1) 
records = []
for _ in range(number_of_samples):
    x = random.randint(0, max_days_out)
    sample_date = start + dt.timedelta(days=x)
    records.append({
        'sample_date': sample_date,
        'y': y_formula(sample_date)
    })
regression_df = pd.DataFrame.from_records(records)<jupyter_output><empty_output><jupyter_text>Review dataframe.<jupyter_code>regression_df.head()<jupyter_output><empty_output><jupyter_text>Visualize the Data Some visualizations. Let's start with the full time period.<jupyter_code>sns.scatterplot(x='sample_date', y='y', data=regression_df)
plt.xlim((start, start + dt.timedelta(days=max_days_out)))<jupyter_output><empty_output><jupyter_text>Then several months.<jupyter_code>sns.scatterplot(x='sample_date', y='y', data=regression_df)
plt.xlim((start, start + dt.timedelta(days=100)))<jupyter_output><empty_output><jupyter_text>Let's look at 3 weeks of data.<jupyter_code>sns.scatterplot(x='sample_date', y='y', data=regression_df)
plt.xlim((start, start + dt.timedelta(days=21)))<jupyter_output><empty_output><jupyter_text>Build Pipeline and Regressor<jupyter_code>date_transformer = ColumnTransformer(
    [('city_category', 
      DateLinear(
          cols=['sample_date'],
          periods=['year', 'dayofweek', 'dayofmonth']),
      ['sample_date'])
    ],
     remainder='drop')
estimators = [('date_transformer', date_transformer), ('lr', LinearRegression())]
pipe = Pipeline(estimators)
X = regression_df[['sample_date']]
y = regression_df[['y']].values
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=42)<jupyter_output><empty_output><jupyter_text>Fit Pipeline<jupyter_code>pipe.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Make Predictions and Evaluate<jupyter_code>y_pred = pipe.predict(X_test)<jupyter_output><empty_output><jupyter_text>Let's look at the coefficients.<jupyter_code>pipe['lr'].coef_<jupyter_output><empty_output><jupyter_text>That's pretty close to our coefficients from the original formula! Let's see how our model predicts against new data.<jupyter_code>np.sqrt(mean_squared_error(y_test, y_pred))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10544707091469681, 'fraction_numerical': 0.014388489208633094, 'mean_word_length': 5.991379310344827, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '146948', 'license': 'MIT', 'n_tokens_mistral': 1719, 'n_tokens_neox': 1624, 'n_words': 385, 'path': 'examples/date_linear_example.ipynb', 'repo_name': 'ccastleberry/sk-autobots', 'size': '4865'}