<jupyter_start><jupyter_text>**Log and Power Transformations** * Create a distribution that's not normal.* Create log-transform it.* Plot the original and transformed distribution for comparison.* Apply a statistical test for normality.<jupyter_code>import numpy as np

np.random.seed(0)
pts = 10000
vals = np.random.lognormal(0, 1.0, pts)
plt.hist(vals, bins=20, density=True)
plt.yscale("log")
plt.ylabel("frequency")
plt.xlabel("value range");<jupyter_output><empty_output><jupyter_text>* Apply the standard normalization to z-scores.* Apply a statistical normality test on one of the transformed distributions.<jupyter_code>from sklearn.preprocessing import minmax_scale
from sklearn.preprocessing import StandardScaler
from scipy.stats import normaltest

vals_mm = minmax_scale(vals)
scaler = StandardScaler()
vals_ss = scaler.fit_transform(vals.reshape(-1, 1))
_, p = normaltest(vals_ss.squeeze())
print(f"significance: {p:.2f}")
_, p = normaltest(vals_mm.squeeze())
print(f"significance: {p:.2f}")<jupyter_output>significance: 0.00<jupyter_text>* A normal-like distribution are not getting from transformation by standard scaling and the minmax transformation. * Plot the original and the standard scaled distribution against each other.* The two distributions look the same except for the scale.<jupyter_code>plt.scatter(vals, vals_ss, alpha=0.3)
plt.ylabel("standard scaled")
plt.xlabel("original");<jupyter_output><empty_output><jupyter_text>* Use a log transformation.<jupyter_code>from statsmodels.stats.diagnostic import kstest_normal

log_transformed = np.log(vals)
_, p = kstest_normal(log_transformed)  # stats.normaltest
print(f"significance: {p:.2f}")<jupyter_output>significance: 0.31<jupyter_text>* Can't reject the null hypothesis (p_values > threshold).<jupyter_code>np.std(log_transformed)
np.mean(log_transformed)<jupyter_output><empty_output><jupyter_text>* A standard deviation close to 1.0 and a mean close to 0.0.* The distribution is similar to normal. * Plot the histogram of the log-transformed distribution.<jupyter_code>plt.hist(log_transformed, bins=20, density=True)
#plt.yscale("log")
plt.ylabel("frequency")
plt.xlabel("value range");<jupyter_output><empty_output><jupyter_text>* Apply Box-Cox transformation.<jupyter_code>from scipy.stats import boxcox
vals_bc = boxcox(vals, 0.0)
_, p = normaltest(vals_bc)
print(f"significance: {p:.2f}")<jupyter_output>significance: 0.46<jupyter_text>* Box-Cox transform is normal-like. * Plot the Box-Cox transformed distribution.<jupyter_code>plt.hist(vals_bc, bins=20, density=True)
plt.ylabel("frequency")
plt.xlabel("value range");<jupyter_output><empty_output><jupyter_text>**Imputation** * Replace missing values with constants or infer probable values given the other features.<jupyter_code>import numpy as np
from sklearn.impute import SimpleImputer
imp_mean = SimpleImputer(missing_values=np.nan, strategy='mean')
imp_mean.fit([[7, 2, 3], [4, np.nan, 6], [10, 5, 9]])
SimpleImputer()
df = [[np.nan, 2, 3], [4, np.nan, 6], [10, np.nan, 9]]
print(imp_mean.transform(df))<jupyter_output>[[ 7.   2.   3. ]
 [ 4.   3.5  6. ]
 [10.   3.5  9. ]]<jupyter_text>* The missing values are replaced with the mean of the columns.  **Derived Date Features**  **Holiday Features** * Load the holidays for the United Kingdom.<jupyter_code>from workalendar.europe.united_kingdom import UnitedKingdom
UnitedKingdom().holidays()<jupyter_output><empty_output><jupyter_text>* Get holidays for other places, for example, California, USA.* Extract lists of holidays, and add custom holidays.<jupyter_code>from typing import List
from dateutil.relativedelta import relativedelta, TH
import datetime
from workalendar.usa import California

def create_custom_holidays(year) -> List:
      custom_holidays = California().holidays()
      custom_holidays.append((
        (datetime.datetime(year, 11, 1) + relativedelta(weekday=TH(+4)) + datetime.timedelta(days=1)).date(),
        "Black Friday"
      ))
      return {k: v for (k, v) in custom_holidays}

custom_holidays = create_custom_holidays(2021)
custom_holidays
def is_holiday(current_date: datetime.date):
    """Determine if we have a holiday."""
    return custom_holidays.get(current_date, False)

today = datetime.date(2021, 4, 11)
is_holiday(today)<jupyter_output><empty_output><jupyter_text>**Date Annotations**<jupyter_code>import calendar

calendar.monthrange(2021, 1)<jupyter_output><empty_output><jupyter_text>* monthrange returns the first weekday of the month and the number of days in amonth for a given year and month. * The day of the week is given as an integer, where Monday is 0 and Sunday is 6.* This means the first weekday of 2021 was a Friday. January 2021 had 31 days.<jupyter_code>from datetime import date
def year_anchor(current_date: datetime.date):
      return (
        (current_date - date(current_date.year, 1, 1)).days,
        (date(current_date.year, 12, 31) - current_date).days,
      )

year_anchor(today)<jupyter_output><empty_output><jupyter_text>* The following function provides the number of days since the end of the previous year and to the end of the current year. * Extract the number of days from the first of the month and to theend of the month.<jupyter_code>def month_anchor(current_date: datetime.date):
      last_day = calendar.monthrange(current_date.year, current_date.month)[0]
    
      return (
        (current_date - datetime.date(current_date.year, current_date.month, 1)).days,
        (current_date - datetime.date(current_date.year, current_date.month, last_day)).days,
      )

month_anchor(today)<jupyter_output><empty_output><jupyter_text>**Paydays** * Most people would get paid on the last Friday of the month.* Write a function for this.<jupyter_code>def get_last_friday(current_date: datetime.date, weekday=calendar.FRIDAY):
      return max(week[weekday]
        for week in calendar.monthcalendar(
            current_date.year, current_date.month
        ))

get_last_friday(today)<jupyter_output><empty_output><jupyter_text>**Seasons** * Get the season for a specific date.<jupyter_code>YEAR = 2021
seasons = [
    ('winter', (date(YEAR,  1,  1),  date(YEAR,  3, 20))),
    ('spring', (date(YEAR,  3, 21),  date(YEAR,  6, 20))),
    ('summer', (date(YEAR,  6, 21),  date(YEAR,  9, 22))),
    ('autumn', (date(YEAR,  9, 23),  date(YEAR, 12, 20))),
    ('winter', (date(YEAR, 12, 21),  date(YEAR, 12, 31)))
]

def is_in_interval(current_date: datetime.date, seasons):
      return next(season for season, (start, end) in seasons
                if start <= current_date.replace(year=YEAR) <= end)
    
is_in_interval(today, seasons)<jupyter_output><empty_output><jupyter_text>**Sun and Moon** * The Astral module offers information about sunrise, moon phases, and more. * Get the hours of sunlight for a given day in London.<jupyter_code>!pip install astral
from astral.sun import sun
from astral import LocationInfo
CITY = LocationInfo("London", "England", "Europe/London", 51.5, -0.116)
def get_sunrise_dusk(current_date: datetime.date, city_name='London'):
      s = sun(CITY.observer, date=current_date)
      sunrise = s['sunrise']
      dusk = s['dusk']
      return (sunrise - dusk).seconds / 3600

get_sunrise_dusk(today)<jupyter_output><empty_output><jupyter_text>**Business Days** * The following function extracts the number of business days and weekends/holidays in a month.<jupyter_code>def get_business_days(current_date: datetime.date):
      last_day = calendar.monthrange(current_date.year, current_date.month)[1]
      rng = pd.date_range(current_date.replace(day=1), periods=last_day, freq='D')
      business_days = pd.bdate_range(rng[0], rng[-1])
      return len(business_days), last_day - len(business_days)

get_business_days(date.today())<jupyter_output><empty_output><jupyter_text>**Automated Feature Extraction** * Featuretools calculates many datetime-related functions.<jupyter_code>import featuretools as ft
from featuretools.primitives import Minute, Hour, Day, Month, Year, Weekday

data = pd.DataFrame(
    {'Time': ['2014-01-01 01:41:50',
              '2014-01-01 02:06:50',
              '2014-01-01 02:31:50',
              '2014-01-01 02:56:50',
              '2014-01-01 03:21:50'],
     'Target': [0, 0, 0, 0, 1]}
)        
data['index'] = data.index
es = ft.EntitySet('My EntitySet')
es.entity_from_dataframe(
    entity_id='main_data_table',
    index='index',
    dataframe=data,
    time_index='Time'
)
fm, features = ft.dfs(
    entityset=es, 
    target_entity='main_data_table', 
    trans_primitives=[Minute, Hour, Day, Month, Year, Weekday]
)
fm<jupyter_output><empty_output><jupyter_text>**ROCKET** * Load an example time-series in SkTime.<jupyter_code>from sktime.datasets import load_arrow_head
from sktime.utils.data_processing import from_nested_to_2d_array
# please note that this import changes in version 0.8:
# from sktime.datatypes._panel._convert import from_nested_to_2d_array

X_train, y_train = load_arrow_head(split="train", return_X_y=True)
from_nested_to_2d_array(X_train).head()
from sktime.transformations.panel.rocket import Rocket
rocket = Rocket(num_kernels=1000)
rocket.fit(X_train)
X_train_transform = rocket.transform(X_train)<jupyter_output><empty_output><jupyter_text>**Shapelets**<jupyter_code>from sktime.transformations.panel.shapelets import ContractedShapeletTransform
shapelets_transform = ContractedShapeletTransform(
    time_contract_in_mins=1,
    num_candidates_to_sample_per_case=10,
    verbose=0,
)
shapelets_transform.fit(X_train, y_train)
X_train_transform = shapelets_transform.transform(X_train)
X_train_transform<jupyter_output><empty_output><jupyter_text>* This gives a transformed dataset that can use in machine learning models.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11722512552515627, 'fraction_numerical': 0.025207500768521365, 'mean_word_length': 5.541554959785523, 'pattern_counts': {'":': 0, '<': 87, '<?xml version=': 0, '>': 87, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '113401', 'license': 'MIT', 'n_tokens_mistral': 3466, 'n_tokens_neox': 3240, 'n_words': 797, 'path': 'Preprocessing TS/Preprocessing TS.ipynb', 'repo_name': 'thirasit/Time-series-analysis-with-ML', 'size': '9759'}